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ABSTRACT 

This document is a guide to the use of alternative 
assessments (surveys, interviews, observation measures, and 
performance samples) in a family literacy project. This resource 
guide focuses on using alternative approaches to identify learners' 
needs, determine learners' ongoing progress in meeting project 
objectives, ascertain overall success of the project, give staff 
immediate and relevant feedback, enable learners to assess their own 
development, and meet guidelines contained in Title VII of the 
Elementary and Secondary Education Act. Chapter 1 contains an 
introduction to family literacy and presents some of the special 
evaluation issues related to implementing literacy programs. Chapter 
2 presents a model for integrating program planning, implementation, 
and evaluation. Chapters 3, 4, and 5 illustrate the three phases of 
assessment and evaluation. Chapter titles and authors are: (l) 
Introdi|ction to Alternative Approaches to Assessment and Evaluation" 
D. Holt); (2) Integrating Program Planning, Implementation, and 
Evaluation" (D. Ramirez); (3) "Initial Assessment: First Step to 
Success (K. Graham); (4) "Assessing Ongoing Progress: Are We 
Progressing?" (H. S. Wrigley) ; and (5) "Collecting, Analyzing, and 
Reporting Alternative Assessment Results" (S. Gelardi) . One figure 
and 29 sample handouts and survey forms are included. (SLD) 
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PREFACE 



If we don't change directions, we will end up where we 
are headed, 

— Chinese proverb 

Assessment and evaluation strategics help staff members and learners 
identify what direction their project is taking. The strategies also provide 
information that can be used in case staff and learners want to change 
course to achieve different results. This resource guide was developed to 
assist staff members of Family English Literacy Programs (FELP) to design 
and use alternative approaches to assessment and evaluation. Most FELP 
piojects use standardized as well as alternative measures. By using a 
variety of approaches, staff can obtain in depth information about the 
learners' backgrounds, the progress the learners are making, and the overall 
effectiveness of tht project. The guide does not contain all of the informa- 
tion needed to design a comprehensive evaluation of a project; rather, it is 
intended as a supplemental resource for creating and using alternative 
approaches. 

Development of the guide began in December, 1990 at a coordination 
meeting of fourteen FELP projects in California. The needs assessment for 
that event indicated that staff considered assessment and evaluation as 
among the most important challenges they faced. FELP staff emphasized 
that they needed suggestions for procedures and instmments that were 
tailored to the learners, curriculum, schedule, and other key characteristics 
of their projects. They stated that although standardized tests provide a 
partial view of their projects, they needed additional strategies for assessing 
language proficiency as well as hard-to-measure outcomes such as cultural 
adaptation, self-confidence, and literacy awareness. 

Since 1990, the authoi"s have continued to work closely with FELP 
staff to create an evaluation g:uide that meets their needs for alternative 
assessment. The authors met frequently with representatives of FELP 
projects and obtained feedback on drafts of their chapters. In general, FELP 
staff stated that the guide should: 

1. Demystify assessment and evaluation. Staff members need to know 
how to make their own decisions about assessment and evaluation, in 
consultation with a professional evaluator. 

2. Present concrete suggestions for designing and using alternative 
assessment instruments and procedures. Staff want infontiation that is 
immediately useful and sample instruments that can be adapted for 
their projects. 

3. Focus on a limited number of assessment approaches. Rather than 
presenting general information about many different alternative 
measures* the authors concentrated on four alternative approaches: 
surveys, interviews, observation measures, and performance samples. 
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The chapters are organized according to the sequence in which alterna- 
tive approaches are likely to be considered and used in a project. Chapter I 
contains an introduction to family literacy and presents some of the special 
evaluation issues related to implementing literacy programs. Next, Chapter 
II presents a model for integrating program planning, implementation, and 
evaluation. Finally, chapters III, IV, and V iUustrate the three phases of 
assessment and evaluation: (1) detennining learners' needs and establishing 
baseline data; (2) docimfienting learners* progress toward meeting project 
objectives; and (3) collecting, analyzing, using, and reporting data obtained 
with alternative approaches. 

The authors consider this document urfinished. It is presented as a 
field test edition, awaiting feedback from individuals who are interested in 
trying out new ways of assessing the success of their projects. The ex- 
amples of alternative assessment instruments presented in the volume are 
not intended as ready-to-use assessment tools; rather, the authors understand 
that FELP staff will modify these tools or create their own instruments and 
approaches based on the unique characteristics of the learners, staff, and 
program design. As the instruments and other infomiation in the volume arc 
adapted to the specific needs of individual PELF projects, assessment and 
evaluation of family literacy projects will become more beneficial to learn- 
ers and staff. 

The authors look forward to working wiUi FELP staff members to 
assess and evaluate the contents of this volume. They invite readers to 
complete the surveys at the end of each chapter and become collaborators in 
the development of this publication. The feedback will be used to make 
revisions for a final edition that will be published in the futurc. 

MARYT. MAHONY 
Office of Bilingual Education 
and Minority Languages Affairs 
US. Department of Education 
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CHAPTER I 

INTRODUCTION 
TO ALTERNATIVE 
APPROACHES TO 
ASSESSMENT AND 
EVALUATION 




Daniel D. Holt 
California Department of Education 
Sacramento, California 



Among the many responsibilities of staff members of a project fxinded under 
the Family English Literacy Program (FELP)* are the tasks of assessing 
and evaluating their project. Such responsibility involves developing a com- 
prehensive evaluation design for determining the degree to which project 
goals and objectives are being met. The purpose of this guide is to assist 
FELP staff witii implementing a key part of effective evaluation— that is, 
designing and implementing alternative approaches to assessment. By inte- 
grating alternative approaches witii standardized measures, FELP staff can 
obtain mor^ accurate and complete information for improving their project. 

This chapter will first present \he distinctions between standardized and 
alternative assessments. Next, Uie special evaluation needs of FELP projects 
wiU be highlighted. Finally, the chapter will provide overviews of chapters 
II-V of the guide. 

* Sec the Glossary on pages 1 14-115 of this publication fordcfmitions of key terms that appear in this 
chapter. 



Alternative approaches 
allow FELP staff to identify what 
is important in their project and 
select assessment strategies 
that are tailored to the unique 
characteristics of the learners 
and the project. 



Standardized and Alternative Assessments 

This resource guide is based on the assumption that successful 
assessment and evaluation depend on using a variety of approaches. 
Some approaches are st^mdardized, such as the Basic English Skills 
Test (BEST) or the Comprehensive Adult Student Assessment System 
(CASAS), In contrast, alternative approaches refer to procedures and 
instruments that are tailored specifically to the learners, curriculum, 
and overall design of the project. In this section, standardized and 
alternative approaches will be described. 

Standardized Approaches 

Standardized approaches rely on instruments that are created to 
allow for comparisons between individuals* current achievement and 
the average performance (nonns) of selected participants (noiming 
group). These approaches offer many advantages to FELP staff; the 
tests can be obtained readily, administered easily, and scored immedi- 
ately and accurately in a cost-effective manner. Staff can use standard- 
ized instruments for comparing the perfomiance of project participants 
with the norms established by the test. 

Standardized approaches, however, provide only part of the 
information needed for documenting learners* progress. For example, 
the results of standardized measures may not be useful or meaningful 
to staff members if the content of the tests is not related to the goals 
and curriculum of the project. Also, the tests may measure discrete 
skills like word recognition, but the project may be designed to 
strengthen shared literacy activities between parents and children. In 
addition, standardized tests may be based on a nonfiling group that 
differs from the participants in the project, thereby making it difficult 
to compare the participants* perfomiance with the standard (nonn 
group). 

Because standardized measures usually focus on products like 
msponses to specific test items, they may fail to provide useful infor- 
mation about imporisini processes like enjoying books or obtaining 
needed social services. Furthermore, many teachers may not fully 
understand how to interpret and use the scores. Even when staff are 
fully trained in the use of standardized tests, the scores may not 
provide information that is useful for making decisions about the next 
steps that should be taken for designing activities, selecting instruc- 
tional materials, or refining project goals. 

Alternative Approaches 

Alternative approaches to standardized assessment and evalua- 
tion may be characterized as flexible, representative of the curriculum, 
meaningful .0 learners, and indicative of learners* acquired ability or 
knowledge (Navarrete, Wilde, Nelson, Martinez, and Hargctt, 1990). 
Alternative approaches aUow FELP staff to identify what is important 
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Surveys are used to obtain 
general information from large 
numbers of individuals. 




interviews are designed to 
coMect detailed information. 




Observation measures are 
used for collecting and recording 
information about various aspects 
of the program. 




Performance samples are 
examples of the learners' work in 
selected tasks. 



in their project and select assessment strategies that are tailored to the 
unique characteristics of the learners and the project. Unlike standard- 
ized tests, alternative aj^roaches can identify the learners* strengths 
and needs without compp^ing results with an external standard or 
norming group. The approaches provide multidimensional, highly 
current views of the learners' progress in many different contexts. 
Because the contents of the instruments can be linked directly to the 
curriculum of the project, alternative approaches give staff members 
immediate access to feedback for planning subsequent learning 
activities. 

Four alternative approaches are discussed in this volume: 
surveys, interviews, observation measures, and performance samples. 
To provide a consistent analysis in all of the chapters, the authors 
adopted definitions of the approaches that may differ somewhat from 
the way in which they are discussed elsewhere in the assessment 
literature. The following is a description of the four alternative 
approaches presented in this guide, 

Surveys arc used to obtain general information from large 
numbers of individuals. Questions used in surveys may be open- 
ended, allowing subjects to provide their own responses, or closed- 
ended, requiring subjects to select only from the response choices 
provided. The answers may be recorded by the respondents or by the 
person(s) who administers the survey. Examples of surveys are 
presenteu in Chapter III. 

Interviews are designed to collect detailed information. The 
interviewer asks questions orally and may make follow-up inquiries to 
clarify or amplify responses. The interviewer may document the 
results by taking notes or tape-recording responses (Bell, 1988). 
Although interviews are usually administered individually, they also 
may be conducted with groups of people. Chapters III and V describe 
how interviews can be used with focus groups. 

Observation measures are used for collecting and recording 
information about aspects of the project such as learner characteris- 
tics, group interactions, or literacy performance. They may be used for 
closed-ended assessments, such as indicating the learner's level of 
listening comprehension; or they may be more open-ended in which a 
staff member judges how confident a student appears when participat- 
ing in a cooperative learning group. Chapter IV provides examples of 
observation measures. 

Performance samples are examples of the loamers* work in 
selected tasks. For instance, learners may be asked to write a story, 
give an oral presentation, participate in a role play, or read a poem. 
Staff members then determine how the task will be documented and 
how the learner's ability to carry out the task will be assessed. For 
example, stories could be collected in written form, role plays could 
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Chapter! ^ 



lij^il The authors present ex- 
amples of four alternative ap- 
proaches with the understanding 
that project staff members will 
modify the examples or create 
their own instruments and ap- 
proaches based on the unique 
characteristics of the learners, 
staff, and program design. 



Any assessment approach 
has limitations, but by using a 
variety of instruments, staff can 
help ensure that they will obtain 
a comprehensive view of their 
project. 



be video-taped, and poem readings could be audio-taped. Performance 
samples differ from observation measures in that observation measures 
record only the assessment of learners* behavior or performance. 
Performance samples include the learner's actual work and the results 
of the assessment. Examples of performance samples can be found in 
Chapter IV. 

A variety of techniques may be used for recording the informa- 
tion that is collected in each of the four alternative assessments. For 
example, a checklist might be used for indicating responses made to 
questions in a survey. A frequency count could be taken of the an- 
swers obtained in an interview. A rating scale might be used for 
recording judgements in an observation measure. 

Portfolios could be used for collecting and organizing samples of 
learners* writing, results of surveys and individual interviews, infor- 
mation collected through observations, and other examples of learners' 
accomplishments. What is included in the portfolio depends on the 
objectives of the project. Staff can make the contents highly personal- 
ized and meaningful by asking the learner to help decide what is 
placed in the portfolio. 

The authors present examples of four alternative approaches with 
the understanding that project staff members will modify the examples 
or create their own instruments and approaches based on the unique 
characteristics of the learners, staff, and program design. Project staff 
should consider many factors when selecting an alternative approach. 
For example, open-ended responses may give more flexibility to 
respondents but take more time and energy for staff to anaJyze. 
Observation measures are useful for documenting staffs judgements 
but may yield different data depending on which staff member is 
doing the observing. Any assessment approach has limitations, but by 
using a variety of instruments, staff can help ensure that they will 
obtain a comprehensive view of their project. They may find it helpfu.i 
to seek the assistance of an experienced evaluator for designing arid 
implementing alternative assessment approaches. An evaluator can 
help staff create instruments that are psychometrically sound, analyze 
the results of the assessments, and make recommendations for improv- 
ing the project. 

This resource guide focuses on using altemadve approaches to 
(1) identify the learners' needs; (2) determine the learners' ongoing 
progress in meeting project objectives; (3) ascertain the overall 
success of the project; (4) give taff immediate and relevant feedback; 
(5) enable learners to assess thei: own development; and (6) meet the 
evaluation guidelines contained in Title VII of the Elementary and 
Secondary Education Act (ESEA). The authors assume that staff 
members will implement alternative approaches in conjunction with 
standardized measures. By creating evaluation designs that integrate 
alternative and standardized approaches, FELP staff will have more 
information for making decisions about their projects. 
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Role of Evaluation 



An effective evaluation design provides learners and staff 
members with accurate and useful information for designing, 
modifying, and improving their project. Such information also 
enables the staff to me^t the evaluation requirements in the 
ESEA, Title VII, guidelines. In order to create effective assess- 
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Objectives should be clear 
statements that represent a 
range of possible outcomes that 
are reHned as the project is 
implemented. 
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This volume contains suggestions for how objec- 
tives can be used to help FELP project staff define the 
outcomes or changes that are expected as a result of 
the instructional program. The content of objectives is 
determined by what the learners need, what the staff 
members are prepared to teach, and what is possible to 
achieve within the time and budget constraints of the 
project. 

During the initial planning of a project, staff 
members can use the development of objectives as a 
way to focus the direction of instruction based on the 
results of the needs assessment. Fo" example, will the 
project concentrate on listening, speaking, reading, or 
writing skills— or some combination? To what extent will 
the le'^rners' native language abilities be developed? 
How will the project helj. parents support their children's 
school achievement? 

Some objectives may deal with clearly observable 
outcomes like reading and writing; others may specify 
changes that are PDore difficult to see such as self- 
confidence, group participation, or literacy awareness. 
Whatever the outcome, an objective should include 
assessment procedures and instruments that can be 
used to measure the extent to which the objective is 
t>eing achieved. 

Objectives should not be immutable, rigid direc- 
tives from the staff to the learners. Nor should they be 
developed by staff without involving the learners or 
assessing their needs. Rather, objectives should be 
clear statements that represent a range of possible 
outcomes that are refined as the project is imple- 
mented. Among the myriad literacy-related activities 
that a project might address, objectives specify where 
the staff and leamers could begin their collaborative 
effort toward buiWing a successful project. 
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Evaluation is an ongoing, 
colEaborative effort by the learn- 
ers, staff members, and evaluator 
to clarify learners' needs, refine 
project goals, design curriculum 
content, and develop instruc* 
tionai methodology. 



menl and evaluation procedures, staff and learners need to (1) clarify 
the goals and objectives of the project; (2) develop indicators of 
progress in attaining the goals and objectives; and (3) identify the 
information they need to coUect to determine the degree to which 
success has been achieved. 

The focus of the evaluation should be based on the goals and 
objectives that learners and staff members have established. For 
example, if the learners are composing stories that they will read to 
their children, the evaluation design should include the procedures for 
collecting information about the learners* compositions and the 
strategies they used for sharing them with their children. 

Assessment versus Evaluation 

In this volume, assessment refers to the use of instruments and 
procedures to gather data on a regular basis. Assessment may focus on 
identifying learners* needs, documenting the learners* progress toward 
meeting their own goals, and ascertaining the extent to which the 
project objectives are being met. 

Evaluation, on the other hand, refers to tlie integration and 
analysis of assessment data at a given point in time for such purposes 
as (1) interpreting the learners* needs; (2) developing goals and 
objectives; (3) designing the content of the curriculum; (4) selecting 
instructional approaches; (5) monitoring the implementation of the 
project; (6) identifying obstacles to achieving objectives; and (7) 
determining the overall success of the project. An example of an 
evaluation activity is tlie annual evaluation report that is required of 
FELP and other ESEA, TiUe VII, projects by the Office of Bilingual 
Education and Minority Languages Affairs (OBEMLA), U.S. Depart- 
ment of Education. 

This guide presents evaluation as part of a comprehensive 
process of planning, implementing, and improving a project (see 
Chapter II for a description of the planning process). Evaluation 
should not be an isolated event that is conducted at the end of each 
year. Rather, evaluation is an ongoing, collaborative effort by the 
learners, staff members, and evaluator to clarify learners* needs, refine 
project goals, design curriculum content, and develop instructional 
methodology. The essence of the evaluation is the participants* 
specification of what constitutes success in their project, what ap- 
proaches they will use to assess the level of success that they have 
achieved, and how evaluation results will be used to improve the 
project. 

Family English Literacy in ESEA, Title VII 

Family English Literacy became part of ESE.\, Title VII in 
1984. This section outlines the services to be provided according to 
the ESEA, Title VII, statute and some examples of how individual 
projects have been designed. 
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Provisions of the Statute 




FELP projects may be 
located in large cities or rural 
areas, operated during the day or 
evening, or conducted in schools, 
community centers, or the 
participants' homes. 



The Family English Literacy Program is ftmded under Title ^^11 
(Bilingual Education) of the Elementary and Secondary Education 
Act. According to the federal statute: 

The term, * family English literacy* means a program of instruction 
designed to help limited-English-proficient adults and out-of -school 
youth achieve competence in the Engl^^ :;i:.-';uage. Such programs of 
instruction may be conducted exclusive l nglish or in English 
and the student's native language. Where appropriate, such programs 
may include instruction on how parents and family members can 
facilitate the educational achievement of limiied-English-proficient 
children [20USC 3283, Sec.7003(a)(7)]. 

Local educational agencies, institutions of higher education, and 
private nonprofit organizations are eligible toapply for FELP funds. 
Projects are funded for three years. FELP projects are administered by 
OBEMLA, U.S. Department of Education. 

Divei^tty in FELP Projects 

Participants, location, and schedule, FELP projects are charac- 
terized by diversity in terms of participants, location, and schedule. 
Many different groups are served such as American Indian, Cambo- 
dian, Chinese, Haitian, Hispanic, Hmong, Korean, Russian, Samoan, 
and Viemamese. The ages of participants also vary among projects. 
Some projects focus on young adults who are not in school (out- of- 
school youth). Others enroll only adults, some of whom may be 
parents. Still other FELP projects include opportunities for parents and 
children to learn together in intergenerational activities. Projects may 
be located in large cities or rural areas, operated during the day or 
evening, or conducted in schools, community centers, or the partici- 
pants' homes. 

Program design. The projects also differ in the way in which 
they have responded to the program requirements of the ESEA, Title 
VII, statute. All FELP projects include activities that promote the 
learners' acquisition of English. Some projects may focus on language 
development, while others integrate language and life skills. Other 
projects focus on oral commimication as a prelude to literacy while 
others address explicitly the development of reading and writing. 
Some projects include activities to help learners develop their abilities 
in English and their native languages. Other projects assist participants 
to improve their knowledge of topics such as mathematics, nutrition, 
the U.S. educational system, and social services. Subject matter may 
be taught through English or a combination of English and the learn- 
ers' native language. 

Home-school linkage. The projects are further distinguished by 
the degree to which they **include instruction on how parents and fam- 
ily members can facilitate the educational achievement of limited- 
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The special characteristics 
of FELP projects necessitate the 
use of approaches to assessment 
and evaluation that are specifi- 
cally tailored to the needs of FELP 
learners and staff. 



Learners come to the 
FELP project with different goals 
and needs for acquiring language 
and literacy. 



English-proficient children" (see FELP definition at the beginning of 
this section). Some projects attempt to create a close link between the 
parents' learning activities and the educational experiences of their 
children. For example, a project might be designed to help parents 
learn skills that would enable them to serve as volunteers at school or 
to work with their children at home in school-related activities. 

Evaluation Issues in FELP Projects 

The special characteristics of FELP projects necessitate the use 
of approaches to assessment and evaluation that are specifically tai- 
lored to the needs of FELP learners and staff. Alternative approaches 
yield information that learners and staff can use to set project goals, 
decide on instructional priorities, document successes and failures, 
and improve future activities. In December, 1990, staff members 
from FELP projects in California held a coordination meeting in 
which they analyzed the unique aspects of family literacy projects 
and considered innovative approaches to evaluation that would meet 
tiieir needs. This section is based in pan on the discussions at the 
seminar. (See Proceedings from the Family English Literacy Semi- 
nar, 1991, for a summary of the results of the seminar.) 

Enrollment and Attendance 

Unlike students in elementary and secondary schools, learners in 
FELP projects enroll voluntarily. However, before they can enroll and 
benefit from FELP services, they must know that a project exists. 
Because participants may enroll in a project at any time, staff mem- 
bers must be prepared to collect baseline data whenever new partici- 
pants enroll. After enrollment, many adult learners may have diffi- 
culty attending project activities because of scheduling conflicts 
caused by personal, family, or job-related issues. 

Assessment and evaluation strategies need to take into account 
the uneven enrollment and attendance patterns that characterize many 
FELP projects. Alternative approaches, used on an ongoing basis, are 
more effective than infrequent pre- and post-measures for enabling 
FELP staff to identify obstacles to the learners' participation and 
make adjustments that will facilitate enrollment and attendance. 

Multiple Contexts, Goals, and Needs 

Adult learners face demands in tiie home, community, school, 
and work place. Depending on the requirements of these contexts, 
learners come to the FELP project with different goals and needs for 
acquiring language and literacy. Some adults may want to learn En- 
glish to work in a restaurant. Others may be interested in opening a 
business. Some may need to learn English for a job interview, while 
others may want to help their children write folk tales in their native 
language. Still others may have more general goals such as learning 
English to communicate and negotiate with native speakers. 
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'^llji^ Families differ according 
to the purposes for which they 
use their native language and 
English, and the degree to which 
they depend on each language 
to carry oiit taily tasks. 



Adult learners also have different levels of knowledge about the 
school, community, and work place. Staff members need assessment 
instruments to obtain accurate information about the learners' needs in 
order to plan appropriate instmction. Alternative approaches, which 
can be tailored specifically to the learners' language and cultural 
backgrounds, provide staff with highly specialized and useful informa- 
tion. Staff members can use the assessment results for designing 
objectives and selecting instructional content and methodology that 
will validate the learners' prior experiences and facilitate their success 
in the project. 

Literacy and Billteracy 

Most participants in FELP projects have a native language other 
than English. However, families differ according to the purposes for 
which they use their native language and English, and the degree to 
which they depend on each language to carry out daily tasks. For 
example, in some families, spouses may use their native language with 
each other but use a mixture of English and their native language with 
their children. Learners also have different levels of proficiency and 
educational experiences in their native language. Some learners may 
have received a high school diploma, while others may have come 
from countries in which no formal schooling was available. 

According to Quintero and Huerta-Macias (1990), to provide 
appropriate instruction, staff need to know the parents' and children's 
proficiency levels and uses of their native language and English. The 
lack of standardized instruments for assessing proficiency in languages 
other than English makes :he use of alternative approaches essential 
for determining the participants' needs, competencies, and growlli ir 
their native languages. Chapters III and IV contain suggestions for 
assessing native language proficiency. 

Parent and Child Outcomes 

When FELP projects create objectives and instructional activities 
for parents and children, staff members need to monitor changes based 
on the special characteristics of two very different kinds of partici- 
pants. If parents and children are learning together in intergenerational 
activities, the evaluation design needs to examine the effects that 
parents and children have on each other's success in language learn- 
ing, literacy development, academic achievement, and other outcomes 
identified by the projects. Alternative approaches to assessment and 
evaluation allow staff to collect information that can provide answers 
to important questions about the progress that patv;nts and children are 
making in teaching and learning from each other. 

Staff Knowledge 

Staff need to know about the cultural and linguistic backgrounds 
of adult learners to design and implement effective strategies for 
recruiting participants, identifying learners' needs, and designing 
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members and learners will 
improve their access to useful 
and accurate information for 
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appropriate instructional activities. Staff member may need training 
in second language acquisition and bilingual development so that they 
can create effective learning activities for language-minority adult 
learners. 

Because alternative approaches to assessment and evaluation are 
developed in close consultation witli staff and learners, they are 
effective for helping staff improve their knowledge of learners and 
assessing the ongoing progress of the project Chapter 11 emphasizes 
the importance of the development of a literacy framework: lliat is, 
staff members' and learners' shared understanding of the learners' 
lileracy needs, program goals, curriculum and methodology, and 
approaches for assessing and evaluating literacy-related activities. 

Evaluation Requirements in ESEA, Title VII 

Projects that are funded under ESEA, Title VII are guided by the 
evaluation requirements contained in federal program regulations (see 
34 CFR Ch.V [11-1-89 edition]. Sec. 500.50 through 500.51 and 
525.3 l[f]). The requirements apply to the evaluation of all ESEA, 
Title VII, programs and do not have special provisions for assessing 
the adult learners who are the primary participants in FELP projects. 
By reviewing these regulations, staff members can determine how 
they can construct an evaluation design that meets the requirements 
and provides useful information for assessing and evaluating their 
project. The regulations do not prohibit the use of alternative ap- 
proaches to evaluation, nor do they require the use of formal, stan- 
dardized measures. By combining alternative and standardized mea- 
sures, staff members and learners will improve their access to useful 
and accurate information for making decisions about Iheir project. 

In general, the regulations specify that three areas be considered 
in designing the evaluation: (1) student outcome data; (2) technical 
standards; and (3) implementation data. Outcome data refer to what 
the students arc supposed to learn during the project. The students in 
fTELP projects may be adults, parents, parents and their children, or 
out-of-school youth. Project staff need to examine the regulations and 
determine how the required data can be obtained for all of llie partici- 
pants in the project. Chapters 11 through V of this volume emphasize 
the importance of collecting and analyzing outcome data based on the 
special features of the project. Issues regarding the evaluation of 
implementation data that are unrelated to student outcomes (for 
example, staff development) are not addressed in this volume. 

Some of the evaluation requirements contained in the ESEA, 
Title Vll, regulations are presented in the following sections. When 
designing their evaluation, FELP staff should consult the actual 
ESEA, Title Vll, regulations. They also should seek assistance from 
their program officer at OBEMLA, staff of the ESEA, TiUe VII- 
funded Evaluation Assistance Center in their region, and their project 
evaluator. 
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Comparl&on Group 

The regulations require that at least part of a project's evaluation 
design include the assessment of the educational progress of project 
participants against an appropriate non-project comparison group, 
FELP staff members can meet tliis requirement with a standardized test 
by comparing the participants* progress with the test's norming group. 
To complement the results of the standardized test, staff could use 
alternative approaches for assessing the learners' progress witliout 
having to compare their performaiice with a nonprojcct comparison 
group. 

Instead of standardized tests, or in addition to them, FELP staff 
may use alternative assessment approaches with a nonproject compari- 
son group. For example, the current participants' performance can be 
compared with (1) the performance of individuals who are not partici- 
pating in the project and have similar characteristics to the participants 
or (2) baseline data that had been collected at an earlier point in the 
project on the current participants and all other participants who had 
enrolled at an eariier time. FELP staff should note that ESEA, Title 
VII, regulations do not require staff to compare ihe results of every 
assessment in the project with a nonproject comparison group. Qiapter 
V presents suggestions for using alternative approaches with a 
nonproject comparison group. 

Representativeness of the Findings 

Evaluation findings should apply to tlie participants, schools, or 
agencies served by the project. In other word&, the conclusions made in 
an evaluation report should be derived from data on learners served by 
the project and on the full range of services provided by the project. 
Although absenteeism and transiency may prevent staff from obtaining 
assessment data on all learners, data should be collected from a sample 
of learners tiiat is representative of the learners served by the project. 
Similariy, staff should obtain information about the leaniers' perfor- 
mance in the major components of the project such as language devel- 
opment, literacy, parenting, and cultural adaptation. Alternative ap- 
pioaches enable staff to taUor assessment and evaluation to the charac- 
teristics of the learners and the services of the project, thereby helping 
staff to ensure that the evaluation findings are indicative of what is 
actually happening in the project. 

Validity and Reliability 

Alternative instruments and procedures should be valid; that is, 
they should measure what they claim to measure. Assessments also 
should be reliable; in other words, they should produce similar results 
consistently. Staff members need to devote time and resources to 
develop and field test alternative approaches in order to document their 
validity and reliability (see "Guidelines for Developing Reliable and 
Valid Alternative Assessments," 1991). Chapter V addresses issues of 
validity and reliability in the evaluation design. 
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Multiple Measures 

Although not required by ESEA, TiUe VII, regulations, a variety 
of alicmativc approaches should be used for conducting intake and 
initial assessments, monitoring progress, and assessing the overall 
success of the project. Multiple instiiiments and procedures will 
provide staff witii a comprehensive view of the learners, the progress 
ihcy arc making, and the effectiveness of the activities of the project. 

Overview of the Resource GuSde 

This resource guide is intended to assist FELP staff members to 
design and implement alternative approaches to assessing and evaluat- 
ing their project. Each chapter addresses a different aspect of assess- 
ment and evaluation. 

Chapter II contains a model for integrating program planning, 
implementation, and evaluation activities. A central element of a 
successful project is a conceptual framework for literacy that is shared 
among the staff and learners. The framework represents the staffs 
collective thinking regarding the definition of literacy, appropriate 
instructional approaches, the respective roles of staff and learners, and 
other factors that influence the design, implementation, and evaluation 
of the project. The author points out how the program planning and 
evaluation processes operate in synchrony when all participants share 
a common understanding of the design and focus of the project. 

Chapter III addresses initial assessment, including needs assess- 
ment, intake, and placement. The chapter contains examples of 
alternative approaches that have been used successfully for initial 
assessment: surveys, interviews, and writing samples. The author 
suggests strategies for using initial assessments to yield accurate 
baseline information about tlic learners so that staff can design appro- 
priate learning activities and determine the extent to which the learn- 
ers arc making progress in the project. 

Chapter IV describes how alternative assessment and evaluation 
approaches can be used for documenting learners' progress toward 
meeting instructional objectives. Three alternative approaches arc 
presented in detail: observation measures, interviews, and writing 
samples. The author emphasizes the importance of ongoing assess- 
ment for (1) helping staff members determine the extent to which their 
instructional efforts are leading to positive results and (2) showing 
learners that they are making progress toward meeting their own 
goals. 

Chapter V provides suggestions for using surveys, performance 
samples, focus groups, and obsei^^ation measures to demonstrate 
progress toward achieving the goals and objectives of the project. The 
author presents strategics for collecting, analyzing, and reporting data 
obtained with each of tlie alternative approaches. The chapter under- 
scores the importance of staff members' and learners' using the 
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evaluation data for (1) making decisions that will refine the goals, 
objectives, curriculum, and overall design of the project and (2) 
determining the progress made toward attaining the goals of the 
project. 

Summary 

This volume rcsix)nds to the need expressed by FELP staff for 
alternative approaches to assessing and evaluating their projects. No 
single approach to assessment can provide a comprehensive view of 
what is happening in a literacy project. By integrating alternative 
approaches with standardized measures, FELP staff can obtain accu- 
rate and complete infonnation for evaluating their project. Four 
alternative approaches arc discussed in this volume: surveys, inter- 
views, observation measures, and performance samples. Each chapter 
presents strategics for designing and implementing alternative ap- 
proaches that are tailored to the needs of individual FELP projects. 
The chapters follow a sequence that might be used by staff in address- 
ing assessment and evaluation issues: (1) planning the pioj^ct and 
creating the evaluation design; (2) detennining learners' needs and 
establishing baseline data; (3) assessing learners' progress toward 
meeting project objectives; and (4) collecting, analyzing, using, and 
reporting data obtained with alternative approaches. 
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SURVEY OF USAGE 

Chapter I. Introduction to Alternative Approaches to Assessment and Evaluation 

This resource guide must be helpful to you, the FELP director or evaluator, in order to be considered 
a success. In accepting the field test edition of the guide, we hope that you agree to participate in this 
usage survey. We plan to revise the guide based on your comments and criticisms; it then will be 
published for a broader audience. Please share with us your critical comments. After reviewing the 
chapter, use the scale below to indicate whether the guide is written clearly; the scale indicates that 
it is 

1 . Clearly written and understandable, the format is clean; 

2. Generally clear, but some wordings or format could be improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at all clear, not at all understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. In each of the same areas, please use the scale to indicate that the guide has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal; or 

4. Information that I don't need and won't use. 



Rate clarity Area to be rated 

1 2 3 4 Format of the chapter 

1 2 3 4 Wording of the chapter 

Not applicabie The general ideas and information presented 

1 2 3 4 Types of assessments (e.g., alternative, standardized) 

1 2 3 4 Role of evaluation 

1 2 3 4 FELP statute and explanations 

1 2 3 4 Evaluation issues for FELP projects 

1 2 3 4 ESEA, Title VII, evaluation requirements 

1 2 3 4 Overview of the guide 



Rate information 
Notapplicable 
Not applicable 
12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 



Thank you for your assistance. If there are other comments, criticisms, or assistance you would like 
to offer, please feel free to do so; you do not need to list your name. 



«3P Piease return your compieted survey. You may foid, staple, and stamp this page - the address is on the back 
side, if you prefer, you may FAX it to Daniei D. Hoit, Biiinguai Education Office at (916) 657-2928. We also 
would appreciate receiving a copy of any alternative assessments you create as a result of this guide. 



Fold here 




Daniel D. Hoit 

Bilingual Education Office, California Department of Education 

PO Box 944272 
Sacramento, CA 94244-2720 



Fold here 
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CHAPTER II 

INTEGRATING PROGRAM 
PLANNING, IMPLEMENTATION, 
AND EVALUATION 




David Ramirez 
RT International 
Mountain View, California 

"If you ever want anything done right, do it yourself best expresses the theme 
of this chapter. Much has been written regarding the shortcomings and failures 
of traditional assessment* and evaluation to meet the needs of adult literacy 
projects. Current research and practice in adult literacy suggests ways of 
improving the utility and relevance of assessment and research efforts. How- 
ever, the most effective alternatives to traditional approaches are created by 
staff members and adult learners as they collaborate to improve their own 
literacy projects. 

The purpose of this chapter is to describe a process for planning, implement- 
ing, and evaluating literacy programs for limited-English-proficient adults, 
parents, and children. The process emphasizes the importance of using alterna- 
tive approaches for designing and evaluating literacy programs. Although this 
chapter focuses on projects funded under ESEA, Title VII, Family English 
Literacy Programs (FELP), the infomiation could be adapted for other literacy 
projects for language-minority adults. 

The chapter is divided into four sections: (1) an explanation of how assessment 
and evaluation procedures are detemiined in part by how learning is defined; 
(2) a discussion of the requirements for an alternative assessment and evalua- 
tion design; (3) a description of the importance of a well-defined literacy 
framework for the project; and (4) a presentation of the planning process that 
project staff can use as a guide for developing the instructional program and an 
appropriate assessment and evaluation design. 

* Sec the Glossary on pages 1 14-1 15 of this publication for definitions of important terms that appear in this chapter. 
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How learning is defined helps deiemnine how literacy is defined 
and evaluated. The way staff members define learning inlluences what 
assessment information they collect, how they collected it, and how 
they use it in the project Currently there are two views of learning 
with decidedly different implications for data collection and use. The 
first is a traditional view of learning that has dominated educafion 
since the turn of the century. The second view, bolstered by research 
in cognitive psychology, is helping to redefine evaluation. 

Traditional View of Learning 

Wolf, Bixby, Glenn, and Gardner (1991) explain how the field of 
evaluation has come to rely almost exclusively on standardized tests. 
Since the turn of the century, intelligence has been viewed as a fixed 
attribute (unchangeable, like skin color), able to be ranked (some- 
where on a normal curve), and genetically determined. Educators who 
hold this view of intelligence classify students into specific groups 
based on the assumption that they have different abilities and therefore 
need a special curriculum. Such approaches can lead educators to 
assign students permanently to groups according to their perceived 
abilities. Because intelligence is defined as a single attribute that is 
distributed along a confinuum, learning is viewed as one-dimensional, 
with individuals progressing through a series of discrete steps from 
simple entry level skills to full mastery. The percepfion of learning as 
molecular and hierarchical has guided the development and use of 
standardized tests. 

Limitations of standardized tests. "Standardized tests ... arc 
constructed so that only a few can score high; they have yielded 
artifacts rather than authentic samples, promoted an isolated view of 
learning, and treated assessment as a matter of pure measurement" 
(Wolf, Bixby, Glenn, and Gardner, 1991, p. 47). The tests contribute 
to the view that assessment results are end products and that co^ition 
is an individual, isolated acUvity. Test developers have often created 
instruments that are more effective at ranking students than identifying 
where they are developmentally. The ranking of students results in 
various negative consequences, not the least of which is that those 
who rank at or near the bottom are ignored in the learning process or 
otherwise stigmatized. Contemporary research in developmental and 
cognitive psychology, however, has questioned traditional assump- 
tions about learning and suggests decidedly different directions for 
data collection. 

Learning Redefined 

Current research suggests that intelligence is not a fixed phenom- 
enon, but is a dynamic, global capacity. All human acfivity requires 
that "the learner actively develop knowledge by observing, inferring, 
generaUng rules, and building theories" (Wolf, Bixby, Glenn, and 
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Gardner, 1991, p. 48). Learning should be viewed not as the 
development of a series of discrete skills, but as a process 
involving **qualitative and uneven shifts in understanding'' 
(Wolf, Bixby, Glenn, and Gardner, 1991, p. 50). Learning 
represents a series of ever increasing, complex understandings 
as an individual observes, theorizes, tests, observes further, and 
readjusts the theory. 

Learning is not only the acquisition of discrete informa- 
tion; it is also the organization of information as a basis for 
further leaming. Furthermore, learning does not occur in 
isolation. It is affected profoundly by external stimulation from 
other individuals, environmental pressures, or resources. The 
redefinition of leaming suggests that staff members need to 
carefully consider the kind of evaluation data that is collected 
and the methods of gathering data. Assessment procedures 
should be used to determine the learner's ability to construct 
and apply knowledge for a variety of purposes. 

Importance of a Literacy Framework 

Assessment, evaluation, and other components of a 
literacy project need to work together in a mutually supportive 
manner. One way to coordinate the components is to create a 
literacy framework. A project's literacy framework may be 
described as the staff members' and learners' shared under- 
standing of learners' literacy needs, program goals, curriculum 
and methodology, and approaches for assessing and evaluating 
literacy development. 

To design and implement alternative assessment instru- 
ments, staff need to identify outcomes and changes that consti- 
tute the learners' success in the project. In other words, how will 
the staff and learners know that progress is being made in 
attaining the objectives of the project? The staff members' and 
learners' shared understanding of the nature of literacy and the 
specific outcomes that the participants are working toward 
represents a key part of the project's literacy framework. 

As was pointed out in Chapter I, FELP projects have 
different program designs. Some projects may help parents 
improve their individual language abilities, while others focus 
on shared literacy activities between parents and their children. 
Developing a literacy frameworic is one of the first steps that the 
staff and adult learners can take to create a common understand- 
ing of the role of literacy in the project and in the learners' lives. 



A Collaborative Approach to 
Planning and Evaluation 

One of the keys to linking the literacy framework, 
program design, and assessment and evaluation procedures used 
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in a project is a collaborative approach to planning and evaluation. 
This section presents a rationale and process for creating a literacy 
framework for the project and describes the steps involved in a 
collaborative approach to planning and evaluation using examples 
taken from actual FELP projects in California. 

Contexts for Literacy 

Individuals and language groups differ according to the ways in 
which they use language and literacy and the settings or contexts in 
which they read and write (Lytle, Belzer, Schultz, and Vannozzi, 
1989). Thus, adult leamers have different goals and needs for literacy 
development. Learners may need to be able to write notes to their 
children's teacher, interpret monthly bills and bus sci;edules, decipher 
product labels, help their children with school woilc, read a repair 
manual, or complete an application for admission to a training pro- 
gram. Alternative assessments offer staff the advantage of being able 
to measure the learners' ability to use literacy in contexts that are 
relevant to their daily lives. 

Literacy is best viewed as a range of practices or activities 
(Scribner, 1987). In a recent research study conducted by Wrigley and 
Guth (1992, p. 7), ESL literacy instruction was defined as "supporting 
language-minority adults with little English and little formal education 
in their efforts to understand and use English in its many forms (oral 
and written, including prose, document, and quantitative literacy), in a 
variety of contexts (family, community, school, work), so that they 
can reach their fullest potential and achieve their own goals (personal, 
professional, academic)." Chapter IV provides detailed suggestions for 
how curriculum and assessment can be matched with this holistic view 
of literacy. 

Project staff need to work closely with adult leamers to design 
assessment procedures that will identify the types of reading and 
writing the leamers are doing and their individual "purposes and 
aspirations" (LyUe, Belzer, Schultz, and Vannozzi, 1989, p. 55). The 
procedures should help staff and leamers examine their own views of 
leaming, literacy, and how literacy is used in their daily lives. Staff 
and leamers also can collaborate in designing and implementing 
evaluation activities including analyzing and interpreting evaluation 
data. Auerbach (1990) emphasizes that adult leamers bring to die 
project their own goals, prior literacy experiences, and attitudes 
towards reading, writing, and leaming, all of which interact and affect 
what and how they will leam. Such background information needs to 
be used to create meaningful assessment and evaluation activities in 
the project, 

LyUe, Belzer, Schultz, and Vannozzi (1989) designed a frame- 
work to guide the development of assessments and evaluations that 
consider the diverse needs and contexts of adult learr.ers. Their 
framework has four dimensions: 1) literacy practices; 2) reading, 
writing, and leaming strategies arxl interests; 3) perceptions of reading 



ERIC 




Ilp^ Collaboration among staff 
and adult learners in developing a 
literacy framework enables all 
participants in the project to 
increase their understanding of 
Hteracy and improve their ability 
to use literacy to achieve their 
own goals. 



One of the assumptions 
of an effective literacy frame- 
work is that aduK learners ac- 
quire abilities and knowledge in a 
spiral versus sequential fashion, 
with increasing levels of profi- 
ciency and understanding emerg- 
ing over time. 



and writing, teaching and learning; and 4) goals. Literacy practices or 
activities refer to how, when, and why learners use literacy in their 
lives. After the staff and learners have identified the role of literacy in 
various contexts, assessment and evaluation approaches can be used to 
measure changes in the way learners use literacy as the project is 
implemented. 

The frameworic outlined by Lytle, Belzer, Schultz, and Vannozzi 
(1989) may be helpftil to FELP participants who are designing their 
own literacy frameworic. For example, the literacy frameworic should 
include a definition of literacy; an identification of the role that 
literacy plays in learners' lives in various contexts; documentation of 
the learners' goals for literacy development; the focus of curriculum 
content and activities; and the basic direction of the assessment and 
evaluation procedures. Collaboration among staff and adult learners in 
developing a literacy framework enables all participants in the project 
to increase their understanding of literacy and improve their ability to 
use literacy to achieve their own goals. 

The next section presents suggestions for designing a literacy 
framework that is based on the learners' needs. The literacy frame- 
woric and planning process contain elements that were borrowed from 
the work of Lytle, Auerbach, and others. The information is presented 
to encourage projects to begin the process of creating leamer-centered, 
participatory approaches to literacy development. The reader should 
understand, however, that this chapter and others in the volume 
contain suggestions, such as developing project objectives and using 
ready-made curriculum, that some advocates of participatory literacy 
approaches might consider inconsistent with authentic learner-centered 
programs. 

Developing A Literacy Framework Through Collaborative Planning 

The development of a literacy framework is the first step in 
designing alternative assessment and evaluation procedures. The 
literacy frameworic is comparable to model curriculum standards that 
educators can use as a foundation for developing programs that meet 
local needs (see Model Standards for Adult English-as-a-Second 
Language Programs, 1992). 

One of the assumptions of an effective literacy frameworic is that 
adult learners acquire abilities and knowledge in a spiral versus 
sequential fashion, with increasing levels of proficiency and under- 
standing emerging over time. As staff and learners woric together to 
create the frameworic and related curriculum, they also can develop 
project-based assessment and evaluation procedures for documenting 
progress toward meeting objectives and making improvements in the 
program. The importance of the literacy framework as a foundation for 
assessment is echoed by Lytle, Marmor, and Penner (1986, p. 22): 
**Our starting point was the understanding that assessment must 
correspond to program philosophy and goals, since assessment proce- 
dures embody and thus convey particular concepts about literacy." 
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At a meeting of selected California FELP projects, staff members 
attempted to reach consensus on a definition of literacy. The results of 
their discussion are summarized in the seven points listed below 
C*Results of the Focus Group Meeting/* 1991, p. 4). 

1. Literacy is finding meaning from print in order to fulfill the learners* 
academic, psychological, political, and social goals. 

2. Literacy includes a range of activities or practices in multiple contexts 
wiih a variety of people. It may involve the use of more than one 
language. 

3. Contexts differ with respect to the complexity of literacy demands. 

4. Literacy activities include reading, writing, and learning strategies. 

5. Literacy in more than one language strengthens learners* linguistic and 
cognitive development and increases their sociocultural knowledge. 

6. The development of literacy is influenced by the language, culture, 
status, experiences, and perceptions of learners and teachers. 

7. The concept of literacy is based on the teachers* and learners* values 
and beliefs. 

The seven statements reflect the staff members' underlying 
assumptions about literacy. From this initial frameworic, several 
implications can be drawn. First, literacy should be developed in ways 
that are meaningful to learners. Second, the literacy curriculum needs 
to address the ways in which learners want to use literacy. Third, 
learners should be deeply involved in the development of the curricu- 
lum content. Fourth, literacy instruction is a collaborative effort 
between the learner and instructor. 

In an actual project, the literacy framework would be refined as 
the project is implemented and evaluated. Initial asj)ccts of the frame- 
woric might be quite general; however, as the staff and learners 
continue to work together, they will improve their understanding of 
the nature of literacy and how it can be fostered in the project. 

To support a literacy framework like the one described above, 
assessment and evaluation procedures should: 

1 . Focus on the learners — that is, accurately identify leanaers* 
strengths and needs and assess learners* progress in the project; 

2. Document how improvement in learners* literacy development 
affects their children, work responsibilities, and community activi- 
ties; 

3. Rely on a variety of instruments and strategies for collecting data; 

4. Reflect changes in learners* perceptions, attitudes, and behavior 
inside d^u^ outside the classroom; 

5. Collect and analyze data on a continuous basis throughout the 
project. 

Process and Product Evaluation 

Alternative assessment and evaluation procedures raise a number 
of questions related to process and product evaluation. A process 
evaluation focuses on how a project is implemented. A question that 
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might be addressed in a process evaluation would be the following: 
How and to what extent are learners involved in identifying their 
literacy goals and objectives, designing the learning activities of the 
project, and assessing their progress in attaining their goals? 

A product evaluation is concerned with the effect of the project 
on the adult learners. A product evaluation addresses such questions 
as: To what extent have adult learners increased their language profi- 
ciency? To what degree have they attained their personal literacy 
goals? To what extent does increased literacy development affect how 
learners (1) perceive themselves; (2) relate to their spouses or other 
family members; (3) respond to friends and co-workers; and/or (4) use 
their literacy skills in new settings or different purposes? 

To answer these and other questions, learners need to be assessed 
in diferent ways. The alternative approaches presented in chapters III, 
IV, and V include surveys, interviews, observation measures, and 
performance samples. Surveys and interviews of family members, 
friends, co-workers, employers, and classmates can provide valuable 
information regarding learners' needs, goals, and preferences for 
curriculum content. Observation measures can be used by staff to 
assess the learners' progress in acquiring language and literacy. 
Performance samples of learners' work, such as writing samples, tape 
recordings of oral readings, and joumals, are useful for documenting 
learners' language development. By using a variety of alternative 
approaches, staff and learners can develop a comprehensive picture of 
what is happening in the project and satisfy the needs of both process 
and product evaluation. 

A Planning Process Model 

Developing a quality instructional program and evaluation design 
requires a thorough planning process involving the collaboration of 
staff members and adult learners. Figure 1 illustrates each of the steps 
of the process. The model is heuristic; that is, it is shown as an ideal- 
ized guide for planning. In actual practice, the tasks are not always 
implemented in sequence. Sometimes steps are skipped and then gone 
back to. At other times, steps are addressed simultaneously. Most 
importantly, the steps are interdependent; that is, what happens in 
implementing one step influences activities in other steps. 

The planning process is cyclical; the knowledge gained by 
implementing one step is used to inform decisions related to other 
steps. The implementation of thie steps should be a collaborative effort 
between staff and adult learners. Figure 1 illustrates that effective 
planning is not a one-time effort, but rather a series of activities that 
occurs throughout the life of the project. By continuously examining 
the needs of the learners, staff, and the community in general, staff 
members can create instructional activities, assessment strategies, and 
evaluation procedures that are highly responsive to the learners and the 
conditions under which the project is being implemented. 
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ries of activities that occurs 
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Tne steps in the model take time for staff and learners to organize 
and implement. Although the entire process cannot be implemented 
fully in any one year, staff and learners should work through as many 
steps as possible, implementing fully when feasible, and laying the 
groundwork for other steps that might be addressed later. 

The process is designed to give staff and learners a blueprint for 
continuously collaborating to design and refine their project. Visualiz- 
ing the entire process enables staff to schedule time and resources for 
dealing with each step. For example, teachers may devote class time to 
activities in which learners would discuss the relevance of instruc- 
tional content to their personal goals and the extent to which they 
believe they are making progress. Ideally, staff members should be 
able to meet weekly or biweekly to assess and clarify goals, objec- 
tives, instructional activities, and other issues. 

Staff members will benefit from planning activities that result in 
(1) clarifying adult learners' needs; (2) defining the dimensions of 
literacy addressed by the project; (3) refining curriculum content; and 
(4) selecting appropriate instructional methods. By establishing a 
consensus about the important aspects of the project, staff members 
can coordinate their responsibilities more successfully. They also can 
provide a clear orientation to new staff members and adult learners 
who join the project in midstream. A literacy framework becomes a 
useful reference for communicating the program's philosophy, goals, 
and procedures to others. Specifying clear goals and objectives helps 
participants collaborate to create a consistent, coherent instructional 
program that responds to learners' needs and supports their continued 
growth. A collaborative planning process enables participants to 
identify changes that will improve the project based on the collective 
resources of staff and l^jamers. 



Positive interpersonal 
relations can be enhanced by 
creating activities designed to 
build a cooperative spirit among 
staff and adult learners. 



Applying the Planning Process Model 

Collaboration is facilitated when individuals establish a trusting, 
supportive relationship with each other. The success of the planning 
process rests on the quality of the relationships among the participants. 
Positive interpersonal relations can be enhanced by creating activities 
designed to build a cooperative spirit among staff and adult learners. 
Team building exercises arc useful for developing participants' ability 
to recognize each person for his/her contribution and realize the value 
of helping and receiving help from one another. Developing coopera- 
tive skills will improve Lhe participants' ability to implement the 
planning process model. (For more on improving human relations in 
the work place, see Johnson, 1987,) Chapter IV contains procedures 
for assessing adult learners' group participation skills. 

The following are descriptions of each step of the planning 
process model and examples of how the step has been addressed by 
FELP projects in California. These descriptions represent only some of 
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ihc many approaches that are possible for designing, implementing, 
and evaluating FELP projects. 

Q Develop the goals of the project. 

Goals are broad, general statements of the intent, purpose, or 
expected outcome of a project. The goals of a FELP project should be 
derived in part from the purposes of the program as delineated in the 
ESEA, Title VII, statute for FELP. (See Chapter I for a description of 
FELP as defined by ESEA, TiUe VIL) 

Goal statements help staff members ^esign the needs assessment 
activities that take place in Step 2 of the planning process. In the initial 
stages, goals are tentative, because they may need to be adjusted based 
on what the staff learn from assessing the needs of the learners. As the 
project is implemented (steps 4-12), the staff may need to adjust 
further the goals of the project. 

The goals developed by the FELP project at the Santa Clara 
County Office of Education (1989-92) address the major purposes of 
FELP; that is, developing the participants* English language and the 
parents* ability to help their children succeed in school. The goals 
were designed to reflect the unique interests of the learners and staff in 
the Santa Qara area {Proceedings from the Family English Literacy 
Seminar, 1991, p. 62). 

1 . Assist LEP Hispanic and Southeast Asian parents and family members 
to achieve literacy and competence in the English language; 

2. Provide parents and family members with knowledge of techniques 
designed to assist their children in developing literacy skills in English 
and appreciation of reading; 

3. Involve parents and family members in direct instructional and support 
roles related to their children's skill development and participation in 
English literacy; and 

4. Support parcnt/community/school communication and the development 
of parents* leadership, advisory, and advocacy roles. 

The goals represent the focus of the project and reflect the 
choices that the project staff have made within the flexibility of the 
FELP statute and the needs of the participants as anticipated by the 
staff. 

Q Conduct the needs assessment* 

Assessing the needs of prospective learners is a vital step in the 
planning process. During this step, staff obtain valuable information 
about the learners, their families, and their commimifies. The first 
phase of the needs assessment is usually conducted as part of the 
proposal for FELP funding. The questions asked of learners are based 
on the goalc tiiat staff intend to pursue in response to authorized 
activities ouUined in ESEA, TiUe VIL Upon funding, staff members 
need to conduct more in-depth needs assessments, the results of which 
may cause staff to clarify or modify the goals. 
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Before the FELP project in the Lincoln Unified School District 
(1990-93) designed its objectives and instructional activities, staff 
administered a survey (alternative assessment instrument) to a repre- 
sentative sample of the Southeast Asian families that the district 
intended to serve. (Chapter III contains a more detailed explanation of 
the needs assessments that were conducted by staff at Lincoln.) The 
results of the survey indicated that the parents wanted services to be 
available near their apartment complex. Tiierefore, the staff estab- 
lished a family education center in a vacant apartment unit located in 
the middle of a two-block area of five apartmexit complexes occupied 
by Cambodian and other Southeast Asian families. 

The staff of the FELP project at the National Council of La Raza 
(1988-91) used writing samples produced by project participants as 
part of its needs assessment. The adult learners wrote the sample in 
Spanish or English, depending on their language proficiency. The staff 
used a holistic technique to score the learners' writing. The samples 
provided important information for refining objectives, developing 
instructional activities, and generating baseline data for comparing and 
evaluating changes in the learners' writing ability at a later time in the 
program. Chapter IV contains more information about using writing 
samples for assessment. 

The FELP project in the Whitticr Union High School District 
(1990-93) used data obtained from the academic records of the adult 
learners' children to design objectives and instructional activities. This 
data enabled staff to identify high school students who were experi- 
encing academic problems and to design project activities that would 
help parents improve their children's academic performance. Informa- 
tion gathered through needs assessment activities can help staff to 
design project objectives (Step 3) and develop curriculum and instruc- 
tional activities (Step 4). 

^ Design instructional objectives. 

Objectives can help staff specify changes or outcomes that 
learners may experience in their language abilities, parenting strate- 
gies, cultural adaptation, or other important areas as a result of the 
instructional program. Some objectives may deal with clearly observ- 
able outcomes like speaking and writing; others, however, may specify 
changes that are more difficult to see such as self-confidence, group 
participation, or literacy awareness. Each objective should include 
procedures and instruments that can t)e used to measure the extent to 
which the objectives are t)eing achieved. (Chapter I contains additional 
information about objectives.) 

Initially, objectives are based on information collected during the 
needs assessment. As the planning process unfolds and more informa- 
tion is collected from the learners, objectives and related assessment 
instruments may need to be refined. Ultimately, objectives are deter- 
mined by what the learners need, what the staff are prepared to teach, 
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and what is possible to achieve within the time and budget constraints 
of the project. 

The following are examples of objectives that use alternative, 
non-standardized assessment measures ("Family English Literacy 
Program: ESEA, TiUe VII Application," 1991, pp. 26-29). 

1 . By the end of each project year, participants will improve their recep- 
tive (listening and reading) and productive (speaking and writing) skills 
at least one level as assessed by an observation measure and writing 
sample. 

2. By the end of each project year, participants will increase by at least 
four the number of literacy activities they do with their children as 
assessed by individual interviews. 



Develop the curriculum and instructional activities. 

Instructional activities are based on the learners' needs and the 
project's goals and objectives. For example, the staff of the FELP 
project in the Parlier Unified School District (1989-92) responded to 
the large numbers of Spanish-speaking families in the district by 
designing activities in which parents use original stories from Latin 
American countries for story-telling activities with their children. 

The FELP staff of the Fremont Unified School District (1988- 
91) organized instructional activities for parents that used children's 
stories from the distnct's adopted list of core literature. By becoming 
aware of the literature that their children studied in school, the parents 
gained confidence in talking with their children about school activi- 
ties. Chapter IV contains m.ore suggestions for developing instruc- 
tional activities and designing ongoing assessment instruments and 
procedures. 



Identify obstacles* 

"Forewarned is foreanmed" accurately dcbc v'ocs the value of 
anticipating potential obstacles to planned activities. Knowing about 
possible impediments helps staff and learners anticipate and refine 
future activities by modifying plans and elaborating strategies to 
overcome hurdles. At a coordination meeting of fourteen California 
FELP projects in December. 1990, participants identified obstacles 
they faced with regard to: 1) recruitment of participants; 2) maintain- 
ing enrollment; and 3) staff recruitment and training. Next, they 
shared strategies they had found successful in addressing these 
problems. The following are results of their discussions {Proceedings 
from the Family English Literacy Seminar, 1991, pp. 15-20). 
1. Obstacles to recruiting participants: 

a. Schools have incomplete or inaccurate lists of limited-English- 
proficient students. 

b. Printed flyers for advertising classes are ineffective because parents 
may not be literate in English or their native language. 

c. Attending school may be a new experience for parents who have 
little or no education in their native countries. 
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d. Parents are reluctant to visit the school because they are not knowl- 
edgeable about the educational system. 

e. Students who qualify for the Greater Avenues for Independence 
(GAIN) program may not participate in the project if classes do not 
meet GAIN requirements. 

f. Some parents face cultural barriers that resoict their participation. 
For example, in some cultures, wives cannot attend class without 
their husband's permission. 

g. Child care is not available. 

2. Obstacles to maintaining enrollment: 

a. Participants must move frequently because of seasonal jobs. 

b. Participants' overtime work prevents them from attending 
classes. 

c. Responsibilities at home prevent regular attendance — for 
example, child care, family problems, and transportation. 

d. Participants may become discouraged if the project activities 
do not meet their needs. 

3. Obstacles to staff recruitment and training: 

a. Staff who are knowledgeable about adult ESL, parenting, and 
language-minority issues are in short supply. 

b. Qualified bilingual teachers are in limited supply. 

c. Some staff are reluctant to work in the evening. 

d. Budgets are insufficient to support staff salaries. 

By identifying obstacles and discussing possible solutions, 
participants are better able to increase the number of alternatives that 
they can consider for overcoming significant barriers to improving 
their projects. 



Identify resources needed for addressing 
each objective and obstacle. 

Staff need to identify the resources necessary for addressing 
objectives and obstacles. Resources include personnel, time, materials, 
and funding. This step helps staff analyze the scope of the project and 
establish realistic expectations for what can be accomplished in the 
Accountability is espe- project. 

ciaily important in collaborative ^ Assign staff members responsibility for designing and 

efforts to ensure that each team implementing each task or activity. 

member contributes his/her fair ^^^^ ^^ff members are responsible for particular tasks, they 

share. become stakeholders who regard their contributions as important 

factors in the success of the project. Accountability is especially 
important in collaborative efforts to ensure that each team member 
contributes his^er fair share. 

^ Establish time line for each task or activity. 

Because FELP projects offer a broad array of services and 
activities over an extended period of time, staff members need to 
specify a lime line to ensure that all steps of the planning process are 
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implemented. Steps 6-8 are essential because they provide the param- 
e.ters for developing each activity. 

Examples of Steps 6-8. 

When staff members of the FELP project located at California 
Stale University, Sacramento (1986-89) identified resources needed 
for improving the project (Step 6), they determined that having their 
own curriculum would significantly facilitate the learners' achieve- 
ment of the project's objectives. During the first year of implementa- 
tion, the staff compiled curriculum content and activities that the 
learners especially liked. Each staff member assumed a specific role 
for designing, compiling, and evaluating the curriculum materials 
(Step 7). The adult learners also had responsibilities such as identify- 
ing content that they believed was important to learn and giving 
feedback regarding the effectiveness of learning activities. After 
deciding on the scope of the curriculum, the staff created a time line 
for developing the curriculum for the project (Step 8). 

During the second year, the staff field tested and refined the 
curriculum materials. By the end of the third year, the curriculum was 
printed and used at all sites in the project. Titled Parenting Curricu- 
lum for Language Minority Parents (Holt, 1988), the text was devel- 
oped in English, Chinese, Hmong, Khmer, Korean, Lao, Spanish, and 
Vietnamese. A teacher's activity guide also was developed. The 
curriculum is an example of how staff can collaborate with learners to 
identify and develop resources that enhance the implementation of the 
project. The success of the effort was dependent on (1) staff members' 
having an intimate knowledge of the adult learners' needs; (2) staff 
and adult learners' sharing responsibilities for tasks; and (3) the staff 
members' adhering to a time line for completing the tasks. 

Parenting Curriculum for Language Minority Parents has also 
been a valuable resource for FELP project staff of the Stocktc n 
Unified School District (1988-91). Staff used the curriculum to 
improve parents' knowledge of such topics as parent-child communi- 
cation, child development, techniques of discipline, and the U.S. 
educational system. Each topic, however, was addressed only after the 
staff and learners agreed that it was important. 

Staff of the FELP project in the Ravenswood City Elementary 
School District (1989-92) found that computers were an important 
resource for enhancing the implementation of the project (Step 6). 
During the project the parents learned to use the same kinds of com- 
puters that their children used at school. The computers became a 
resource for motivating parents to participate in project activities and 
for strengthening the relationship between parents and their children. 

^ Specify expected outcomes. 

During this step of the planning process, staff and learners 
identify the type and amount of change or growth that they expect at 
the end of a given time period. This step represents what the project 
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Staff members and learn- 
ers may want to develop out- 
comes that are short-term (the 
ability to explain one's needs to 
an apartment manager) and long- 
term (obtaining a teaching job). 



The results of the data 
analyses should be reviewed by 
staff and leamers to determine 
the extent to which they accom- 
pliithed what they set out to do. 



participants regard as success in the project Growth can be expressed 
in tenns of academic growth (for example, a specified increase in 
positive ratings on a project-developed reading test); social, cognitive, 
or affective development; organizational skills; or parenting skills. The 
outcome statements are more detailed than the objectives that were 
developed earlier in the process (Step 3). They should sp>ecify the 
assessment measures to be used and the amount of growth that seems 
realistic. Staff members and leamers may want to develop outcomes 
that are short-term (the ability to explain one's needs to an apaitment 
manager) and long-term (obtaining a teaching job). 

Design assessment instruments and procedures 
for collecting data. 

Staff and adult leamers need to identify the data that will docu- 
ment progress toward attaining the outcomes specified in Step 9. For 
example, taking a frequency count of the number of times children 
read to parents per month might be an indicator of parents' encourage- 
ment of children's literacy development. 

Outcomes may be assessed with various types of assessment 
strategies. For instance, an outcome such as growth in listening 
comprehension could be assessed through a single measure such as a 
standardized test, an interview, an observation measure, or perfor- 
mance sample. However, an outcome that deals with reading strategics 
that an adult leanier uses to obtain information might be assessed 
through a combination of an interview, an observation measure, and a 
{performance sample to capture fully all of the strategies that the reader 
might be using. To gather data accurately, it is important that a match 
be established between the type(s) of assessment and the nature of the 
outcome. A variety of alternative procedures for assessing specific 
outcomes are discussed more fully in chapters III, IV, and V. 

^Tl Analyze data. 

Data become meaningful when they are analyzed. Various 
approaches can be used for data analysis. Staff may want to represent 
assessment data in numerical scores so that they can generate percent- 
ages, means, and standard deviations. On the other hand, staff may 
wish to describe learner outcomes by summarizing assessment results 
in narrative form. To interpret and analyze data effectively, staff 
members need to understand thoroughly the outcomes that were 
assessed. Chapter V contains detailed suggestions for analyzing data. 

Formulate conclusions and implications. 

The results of the data analyses should be reviewed by staff and 
leamers to determine the extent to which they accomplished what they 
set out to do. That is, were the objectives of the program attained? 
Were the outcomes specified in the objectives realistic? Were they 
matched to the learners' needs? Should instructional activities be 
changed? Should objectives be modified? This step provides staff and 
leamers with the opportunity to make what are often hard choices 
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Conclusions about the 
outcomes of project activities 
should influence the development 
of ok^ectives that will be pursued 
in the future. 



A planning team that is 
made up of staff members and 
learners will help ensure that 
decisions are based on a thor- 
ough understanding of the needs 
of all participants. 



about the project. What has been successful? What needs to be 
strengthened? What ought to be eliminated? What needs to be 
changed? 

As shown in Figure 1 , the decisions made in Step 12 should be 
used to refine the activities associated with steps 1, 2, 3, and others. 
Conclusions about the outcomes of project activities should influence 
the development of objectives that will be pursued in the future. That 
is, the planning process model represents a cycle of interdependent 
steps. When any aspect of the program is changed, all of the related 
aspects should be reviewed to detemfiine if they also need to be modi- 
fied. Chapter V provides a variety of strategies for using and reporting 
data. 

Designing Alternative Approaches to 
Assessment and Evaluation: 
Wliere Do You Begin? 

The preceding section presented a process for planning, imple- 
menting, and evaluating a FELP project. The model emphasizes the 
importance of collaboration among the staff members and learners for 
creating a successful project. The planning process also underscored 
the value of alternative assessment procedures for helping staff ascer- 
tain the needs of adult learners and the progress they are making in the 
project 

The final section of this chapter presents suggestions for imple- 
menting the planning process model and related alternative approaches 
to assessment and evaluation. Implementing the model involves 
deciding (1) who will be involved in the planning process; (2) how the 
planners will work together; and (3) what issues the planners will 
addresb in assessment and evaluation. 

Members of the Planning Team 

The first and perhaps most important step in initiating a compre- 
hensive planning process is to decide who will be involved in the 
process. A planning team that is made up of staff members and learners 
will help ensure that decisions are based on a thorough understanding 
of the needs of all participants. Tlie project evaluator might also be 
included in the planning team to assist the staff and learners with 
designing the evaluation. 

To the extent possiblCv all instructional staff should have the 
opportunity to participate in the planning process. Community liaisons 
also should participate because of their knowledge of the learners' 
language and cultural backgrounds. When deciding on the content of 
the curriculum and specifying learner outcomes, it may be helpful to 
include suggestions from prospective employers, teachers of the 
participants' children, representatives from local govenimenl agencies 
(for example, employment and social service offices), and staff from 
community colleges and universities. 
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Because of the diversity 
of individuals and the range of 
perspectives that need to be 
Wended into a consensus, it is 
vital that the facilitator of the 
planning team be both willing 
and able to promote a coopera- 
tive spirit among team members. 
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The planning team needs 
to decide what data should be 
collected, analyzed, usedv and 
reported to provide a comprehen- 
sive picture of the impip nenta- 
tion of the project. 



Approaches to Planning 

After the planning team has been formed, team members 
need to consider how they can work together effectively. 
Because of the diversity of individuals and the range of perspec- 
tives that need to be blended into a consensus, it is vital that the 
facilitator of the planning team be both willing and able to 
promote a cooperative spirit among team members. The facilita- 
tor needs to ensure that each team member's contributions are 
invited, expressed, valued, and integrated into the decisions that 
are made in each step of the planning process. 

The facilitator has the responsibility of ensuring that 
meetings and tasks are structured so that the project benefits 
from the experiences of all team members. A collaborative 
approach to planning enables staff and learners to become more 
knowledgeable about each other's needs and the value of 
cooperation itself. According to Freire (1973, p. 48), **acquiring 
literacy does not involve memorizing sentences, words, or 
syllables — lifeless objects unconnected to an existential uni- 
verse — but rather an attitude of creation and re-creation, a self- 
transformation producing a stance of intervention in one's 
context." Collaboration helps team members develop a sense of 
responsibility for helping others and the feeling of satisfaction 
that comes from a successful team effort. The act of planning 
and implementing together can provide significant opportunities 
for learning how to assess what one needs and how to take steps 
to improve one's context, whether it be at home, at school, or at 
work. 

Issues In Assessment and Evaluation 

The primary purpose of the planning team is to take the 
necessary steps for designing, implementing, and evaluating the 
project. This section presents questions that the planning team 
needs to answer related to designing the assessment and evalua- 
tion of the project. 

1. What should be evaluated? 

Staff members need to consider how they will collect data 
for assessing the implementation of the project (process evalua- 
tion) and the effects of the project upon the learners' lives 
(product evaluation). To design the process evaluation, the 
planning team should deal with such issues as the role of the 
staff members and learners in planning, implementing, and 
evaluating the project. Other points to examine might be the 
extent to which the goals of the project are consistent vvith the 
goals of the lear -,rs. The planning team needs to decide what 
data should be collected, analyzed, used, and reported to pro- 
vide a comprehensive picture of the implementation of the 
project. 
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To design the product 
evaluation, the most Important 
point for the planning team to 
address Is the specification of 
what constitutes success in the 
project. 



The planning team needs 
to consider who should be in- 
volved in the evaluation, the role 
of each Individual, and strategies 
for facilitating a cooperative 
effort. 



Product evaluation, on the other hand, deals with issues related 
to the effects of the project on adult learners (and their children if the 
objectives of the project include children). To design the product 
evaluation, the most important point for die planning team to address 
is the specification of what constitutes success in the project. In other 
words, how will Uie staff and learners know the extent to which tJie 
objectives of the project are being met? How will participants deter- 
mine tiiat learners have increased their ability to use language for 
various purposes or their self-confidence in working with others? 
What data will be used to make Uiese determinations? How will die 
data be collected? 

To determine the value of collecting any piece of data, the 
planning team needs to clarify the purpose of the data and analyze 
wheUier the cost of collection is worth the required time and resources. 
For example, will the data be used to improve initial assessment; to 
enhance the curriculum; to show learners the progress they are mak- 
ing; to develop staff members' teaching ability; to document progress 
for outside agencies; or to improve the administration of the project? 

2. When should data collection take place? 

After determining what data will be collected, die planning team 
needs to decide when the data will be gathered. Questions to address 
include: Are the data needed before the learners enroll in the praject? 
How many times should data be collected to show progress? Chapters 
III and IV contain information about the timing of initial and ongoing 
assessment procedures. 

3* How will adult learners, staff members, the project evaluator, 
members of the community, and others be involved in conducting 
the evaluation? 

The platting process model emphasizes the importance of 
collaboration if^helping to ensure an evaluation design that is compre- 
hensive and meaningful. The planning team needs to consider who 
should be involved in the evaluation, the role of each individual, and 
strategies for facilitating a cooperative effort. 

4. What data collection procedures will be used? 

The planning team should decide what procedures will be used 
to collect the data. For example, what data will be gathered via 
alternative instalments such as surveys, interviews, observation 
measures, and performance samples? How will standardized measures 
be used? Factors to consider in these decisions are the comparative 
effectiveness of each procedure in providing accurate and reliable 
data; resources needed for developing the procedures; staff members' 
knowledge or prior experience in using the procedures; time and 
resources necessary for collecting and analyzing the data. 

5. How will staff members know that progress is being made? 

Objectives and expected outcomes of the project represent the 
staff members' criteria for project success. For FELP projects, criteria 
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include a range of outcomes L-uch as gains in listening, speaking, 
reading, and writing abilities; improved self-confidence; enhanced use 
of literacy in different contexts; increased abilities of parents to help 
their children in school; improved involvement of parents in their 
children's school activities; and greater use of community resources. 
Chapter IV provides examples of other learner outcomes and related 
alternative assessment procedures. 

6. How will learners, teachers, administrators, other project staff, 
community representatives, employers, family members, represen- 
tatives from the fund' g agencies, and others become knowledge- 
able about the results of the evaluation? 

The plarming team needs to determine the most effective strate- 
gies for conveying the evaluation results to interested audiences. A 
variety of strategies should be utilized so that each group is provided 
sufficient and comprehensible information about the evaluation. 
Strategies may include oral presentations, written reports, and graphic 
illustrations. Chapter V provides examples of strategies for reporting 
the results of the evaluation. 
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When assessment and 
evaluation are integrated with 
instructBon, they produce results 
that can be used to document 
success and plan improvements 
in the project. 



Summary 

Traditional standardized assessment and evaluation procedures 
provide only some of the answers to the questions that staff and others 
may have about FELP projects. The standards on which traditional 
procedures are based are not always tailored to the needs, goals, and 
characteristics of the diverse adult learners who participate in literacy 
projects. In contrast, alternative approaches to assessment and evalua- 
tion enable project staff to create instruments and procedures that 
provide highly useful and meaningful information for improving their 
project. 

To be effective, alternative approaches should be linked carefully 
to the needs of the adult learners, literacy framework, objectives, and 
curriculum content of the project. Such linkage can be achieved by 
utiliang a planning model that is designed and implemented through 
the collaborative efforts of staff and learners. The collective commit- 
ment of all project participants to a comprehensive planning process 
will help ensure that assessment and evaluation activities are imple- 
mented as an integral part of planning and implementing the project. 
When assessment and evaluation are integrated with instruction, they 
produce results that can be used to document success and plan im- 
provements in the project. 
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Assessing Success: 
FELP Resource Guide 



Chapter 



SURVEY OF USAGE 

Integrating Program Planning, Implementation, and Evaluation 



This resource guide must be helpful to you, the FELP director or evaiuator, in order to be considered 
a success. In accepting the field test edition of the guide, we hope that you agree to participate in this 
usage survey. We plan to revise the guide based on your comments and criticisms; it then will be 
published for a broader audience. Please share with us your critical comments. After reviewing the 
chapter, use the scale below to indicate whether the guide is written clearly; the scale indicates that 
it is 

1 . Clearly written and understandable, the format is clean; 

2. Generally clear, but some wordings or format could be improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at all clear, not at all understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. In each of the same areas, please use the scale to indicate that the guide has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal; or 

4. Information that I don't need and won't use. 



Rate clarity 

12 3 4 
12 3 4 

Not applicable 

12 3 4 
12 3 4 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



12 3 4 
12 3 4 



Area to be rated 

Format of the chapter 

Wording of the chapter 

The general ideas and information presented 

Views of learning 

Literacy framework 

Planning process model: Developing goals 
Conducting needs assessment 
Designing objectives 

Developing curriculum and instructional activities 
Identifying obstacles 
Identifying resources needed 
Assigning responsibilities to individuals 
Establishing timelines for each task or activity 
Specifying expected outcomes 

Designing data collection instruments and procedures 
Analyzing data 

Determining conclusions and implications 
Planning team 

Approaches to planning assessment and evaluation 



Rate information 

Not applicable 
Not applicable 
12 3 4 

12 3 4 

12 3 4 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



12 3 4 



12 3 4 



Thank you for your assistance; please add any other comments you feel appropriate. 



Please return your completed survey. You may fold, staple, and stamp this page - the address is on the back 
side. If you prefer, you may FAX it to Daniel D. Holt, Bilingual Education Office at (916) 657-2928. We also 
would appreciate receiving a copy of any alternative assessments you create as a result of this guide. 
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Fold here 




Daniel D. Holt 

Bilingual Education Office, California Department of Education 

PO Box 944272 
Sacramento, CA 94244-2720 



fold here 
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CHAPTER III 

INITIAL ASSESSMENT: 
FIRST 
STEP 
TO 
SUCCESS 



Kathy Graham 
Lincoln Unified School District 
Stockton, California 



Assessment plays an important role in the successful operation of adult 
literacy projects, from tlie initial to the fmal stages of implementation. 
Initial assessment is particularly significant in literacy projects because of 
the importance of linking instructional services to the learners' needs. 

This chapter presents strategies related to the use of initial assessment to (1) 
identify the needs of potential participants before instruction begins; (2) 
assess the needs of actual project participants in order to refine the goals, 
develop the objectives*, and plan instructional activities; and (3) collect 
baseline data for assessing ongoing progress and evaluating program 
outcomes. Although the chapter focuses on projects funded under ESEA, 
Tide VII, Family English Literacy Programs (FELP), the information could 
be adapted for other literacy projects for language-minority adults. 



♦ See the Glossary on pages 114-1 15 of this publication for definitions of important terms 
that appear in this chapter. 
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The design of a FELP 
project should be based on the 
assessed needs of prospective 
participants. 



Role of Assessment in Planning a Project 

Initial assessment is successful when it results in the development 
of a project that is tailored to the specific needs of project participants. 
For the Lincoln Unified School District, located in Stockton, Califor- 
nia, assessment began when a group of English-as-a-second-language 
CESL) teachers became interested in the needs of Southeast Asian 
elementary and secondary students and their families. Teachers recog- 
nized that parents were desirous of knowing more about their 
children's activities in school. Parents wanted to find out how they 
could help their children with difficulties they faced in school and in 
the community. They expressed an interest in knowing more about the 
cultural differences between the United States and their native coun- 
tries. They wanted their children to develop an understanding of their 
native language and culture. Staff recognized a match between the 
interests of parents and the purposes of a FELP project. The staff 
decided to examine the parents* needs more closely and document 
them for the needs assessment required for a FELP grant proposal. 

Needs Assessment Survey 

A survey is an effective approach for assessing the needs of 
potential project participants. It can be administered at the beginning of 
the project to a large group of individuals in order to create a general 
picture of tlie needs of adults and other family members who may be 
interested in project sers'ices. It may consist of a written questionnaire 
or checklist that is completed by prospective participants or adminis- 
tered orally by staff members to individual respondents. Responses 
may be easier to tally on a checklist, but a questionnaire can give staff 
a more complete understanding of the respondents, especially when 
open-ended questions are used. 

Staff of the Lincoln Unified School District developed a survey 
for collecting basic information from prospective participants regarding 
their need for a literacy project (see Example 1*, "Needs Assessment 
Survey**). The survey addresses such issues as the number of adults and 
children residing in each housing unit; their native language; the 
number of adults who would like additional English instruction; 
preferred location and time of literacy classes; parents* need for child 
care and transportation; desired curriculum topics; self-assessment of 
English language proficiency; the number of years* residence in the 
United States; and the number of children who are enrolled in ESEA, 
Title VII, programs. Staff in other FELP projects should adapt the 
survey shown as Example 1 to fit their own purposes for collecting 
initial information from their prospective participants. 

To get the most comprehensive and valid results, four bilingual 
staff members of the Lincoln Unified School District spent three weeks 
visiting 300 apartment units to administer the survey to individual 
residents. Results of the survey indicated that Cambodians and other 



* Examples of assessment instruments may be found a( the end of this chapter. 
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Related Needs Assessment Activities 



Southeast Asian families were especially interested in services that a 
FELP project could provide. Staff also realized that the parents' 
limited access to transportation made it necessary to have FELP 
classes located r>ear their apartments. This finding would lead eventu- 
ally to the FELP project's establishing a family education center in a 
vacant apartment unit located in the middle of a two-block area of five 
apartment complexes occupied by Cambodian and other Southeast 
Asian families. 

Tlie survey results also helped staff determine the most conve- 
nient times for lileracy classes, the literacy needs of the prospective 
participants, and relatexl needs such as counseling and cultural adapta- 
^ tion. The survey was part of a comprehensive needs assessment 

process that is described in the next section. 

The survey results helped 
staff determine the most conve- 
nient times for literacy classes, ^^^^^ administering the survey, the ESL coordinator in the 

Lincoln Unified School District had discussed the needs of the South- 
the literacy needs of the prospec- ^^^^ ^^.^^ community and the merits of a FELP project with members 
tive participants, and related Qf district's bilingual advisory council. She consulted with them as 

needs such as counseling and she developed the program design and considered their comments 

cultural adaptation. about the components of ihc proposed instmctional program. In 

addition to meeting with the advisory council, she had frequent 
discussions with other district staff, bilingual aides, and parents to 
identify needs and corisider various aspects of the project. This infor- 
mation was useful in developing the questions for the survey. 

To gather information about agencies involved in literacy-related 
activities, the ESL coordinator interviewed individuals in charge of 
community service agencies and visited other FELP projects located in 
northern California. These activities enabled her to determine how 
some of the adults* literacy and social needs were being addressed by 
various service providers in the community and the relative effective- 
ness of the services. The coordinator used this information to identify 
content and methodology for FELP classes and strategies for coordi- 
nating project services wiih other community agencies. 

To corroborate the findings of the needs assessments, the ESL 
coordinator analyzed demographic data and statistical reports regard- 
ing Southeast Asian families. For example, she identified the number 
of Cambodian and other Southeast Asian families living in San 
Joaquin County and the projected number that would be residing in the 
Lincoln Unified School District in the future. In addition, research 
findings on aduli literacy indicated that reading and employment 
levels of Cambodian adulLs were significantly below the national 
averages of other Southeast Asians {Report to the Congress — Refugee 
Resettlement Program, 1989). 

The combined results of surveys, interviews, observations, 
meetings, and reports constituted a thorough assessment of the need 
for a FELP project. The results were used to formulate the general 
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Participants can improve 
their ability to cooperate and 
their knowledge of the issues by 
making a commitment to working 
together and drawing on each 
other's expertise. 



The loeirners' IKeracy 
experiences 'n their native lan- 
guage need \o be an^Jyzed and 
considered when designing the 
instructional program. 



direction of tlie project, including the goals and objectives (for more 
information on objectives, see chapters I and II). Before developing the 
detailed program objectives and instructional design, the staff and 
prospective learners shared ideas about how literacy and literacy- 
related content would be dealt with in the project. In other words, they 
began to develop the project's literacy framework. 

Developing a Literacy Framework 

Chapter II emphasized the importance of a literacy framework — 
that is, the staff mem.bers' and learners' shared understanding of the 
learners' literacy needs, program goals, curriculum and methodology, 
and approaches for assessing and evaluating literacy development. 
Developing consensus on such complex issues is a challenging task, 
and staff and learners may vary in the extent to which they are able or 
willing to collaborate to address them. However, participants can 
improve their ability to cooperate and their knowledge of the issues by 
making a commitment to working together and drawing on each other's 
expertise. 

Some of the issues that could be pursued in developing the 
literacy framework include the following: 

1. Identifying the role tiiatlitcricy plays in the learners' lives; 

2. Refining the goals and objectives of the project; 

3. Designing the curriculum content and instructional methodologies 
(including the role of the native language and English in the 
project); 

4. Establishing the basic direction of assessment and evaluation 
procedures; and 

5. Identifying strategies that will help ensure that the literacy frame- 
work is dynamic, flexible, and responsive to the learners' needs. 

Staff should establish the status, significance, and role of \he 
learners* native language within the literacy framework. The learners' 
literacy experiences in their native language need to be analyzed and 
considered when designing tiie instructional program. Depending on 
the learners* needs, FELP projects may provide learners wi?Ji opportu- 
nities for furLher literacy development in their native language; activi- 
ties in which they acquire challenging content through their native 
language; and strategies for using their native language to support tiieir 
children's development. Initial assessment procedures should be 
designed so that they yield useful information for helping staff incorpo- 
rate the learners' native languages and cultures in their projects. 

An Expanded View of Literacy 

A sound literacy framework is based on an expanded definition of 
literacy that includes the social context in which language and literacy 
are used. Literacy occurs everywhere in the adult learner's world: at 
home, in the work place, at school, and throughout the community. 
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LHeraoy occurs every- 
where in the aduft learner's 
world: at home, in the work 
place, at school, and throughout 
the community. 



From the very beginning 
of a project, appropriate assess- 
ment ensures a project's rel- 
evance and effectiveness in 
meeting the learners' needs. 



Furthermore, the very nature of literacy goes beyond communicating 
meaning through speaking, reading, and writing. As suggested by 
Eisner (1991, p. 11), literacy can be considered "the ability to encode 
or decode meaning in any of the fomis used in a culture to represent 
meaning." Language, according to Eisner, is just one form of literacy; 
meaning is also conveyed through music, dance, and visual fonns. 

The importance of Eisner's expanded view of literacy became 
apparent to the Lincoln Unified School District staff after their FELP 
project was funded. The results of the survey (Example 1) revealed 
that, although the majority of the Cambodian participants never at- 
tended school in their native country and could not read or write in 
their native language (Khmer), they were experts at communicating 
meaning through other media such as painting, music, classical dance, 
woodcarving, and cooking. By recognizing the participants' life-long 
achievements and their unique levels of literacy, staff were more able 
to enhance the learners' self-esteem and validate their individual 
abilities. Consequently, part of the program design for the FELP 
project included activities in which adult learners' shared their areas of 
expertise with others (for example, native cuisine, folk dance, music, 
and sewing). Participation was open to all interested adults and chil- 
dren, including members of the English-speaking mainstream commu- 
nity. (Chapter IV presents other activities in which learners teach and 
leam from each other.) 

Importance of Initial Assessment 
in Implementing a Project 

This section will consider integral role that assessment plays 
in successfully refining the preliminary plans for the project into actual 
instructional activities and cuniculum content. Initial assessment is 
critically important to the implementation of any project because it 
allows project staff to (1) reshape and refine their literacy framework; 
(2) place adult learners' at the appropriate literacy level; and (3) 
establish important baseline data for assessment and evaluation. 

Literacy Framework: A Dynamic Continuum 

A project's literacy frameworic should be synchronized with the 
needs, goals, past experiences, and cultural values of the participants. 
From the very beginning of a project, appropriate assessment ensures a 
project's relevance and effectiveness in meeting the learners' needs. 

After staff begin implementing the FELP project, they can use a 
survey again, this time to obtain more detailed information from 
prospective adult learners. The survey is an effective way for staff 
members to gather information about potential participants and to 
publicize the availability of project services. The results of tlie survey 
can be used to identify curriculum content and align project goals and 
objectives with learners' needs, interests, and past literacy experiences. 
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The administration of the 
survey helped staff and learners 
to begin to build the bond of 
mutual trust and respect that is 
essential to the success of any 
project. 



The intake process lays 
the foundation for establishing a 
good working relationship be- 
tween staff and learners. 



During the initial stages of implementing the FELP project at the 
Lincoln Unified School District, the project teacher and her bilingual 
assistant went to individual apartments to administer the **Parent 
Survey" (Example 2) to prospective participants. In addition to 
providing important data about the participants and the community, 
the survey enabled project staff to develop an in-depth awareness of 
the personal and educational needs and goals of the Cambodian 
immigrants. The Parent Siu^ey contains open-ended questions that are 
designed to elicit more comprehensive responses. The administration 
of the survey helped staff and learners to begin to build the bond of 
mutual trust and resf)ect that is essential to the success of any project. 

intake Assessment: the Personal Interview 

After publicizing its existence through surveys, flyers, meetings, 
rallies, and other promotional activities, staff need to establish an 
intake assessment procedure. This phase of initial assessment is 
designed to help staff collect important background information on 
actual project participants including language proficiency, previous 
educational experiences, and length of residency in the United States. 
Instruments for collecting basic intake information are shown in 
examples 3 and 4. 

Because it is often the learners* first in-depth experience with the 
project, the intake process lays the foundation for establishing a good 
working relationship between staff and learners. Therefore, intake 
procedures need to give learners a positive introduction to the project 
and provide staff with useful information for planning instructional 
activities. 

Most FELP projects no longer rely solely on standardized tests 
for intake assessment. Alternative approaches, such as interviews and 
writing samples, can provide staff with meaningful, highly personal- 
ized insights into the learners' background. The interview is a versatile 
assessment approach for obtaining many kinds of information and can 
range in length from a five-minute informal exchange to an elaborate 
two-hour version of the kind described by Lytle (1989, p. 59) as **an 
array of questions and activities designed to engage adults in self- 
exploration of their literacy practices and abilities." 

Auerbach (1990) developed an abbreviated list of Lytle's inter- 
view questions, dividing them into the following categories: students' 
background, employment, education, conceptions about literacy, 
reading, writing, support systems, and needs. Auerbach emphasizes 
that the interviewer should make the intewiewee feel as comfortable 
as possible. Project staff should be prepared to conduct the interview 
in English or in the interviewee's native language, A bilingual inter- 
viewer can facilitate communication and put the interviewee at ease. 

To create a non-threatening atmosphere for the participant, 
interviewers should avoid taking notes during the interview. Important 
information or evaluative comments should be recorded after the 
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Interviewers may need to 
be trained in the use of effective 
interpersonal communication 
techniques so that a positive 
climate is established for the 
interview. 



Project staff need to 
make assessments as non-threat- 
ening as possible because some 
adult learners may not have par- 
ticipated extensively in formal 
educational activities. 



interview is concluded. Interviewers may need to be trained in the use 
of effective interpersonal communication techniques so that a positive 
cHmate is established for the interview. Chapter V contains more 
information on the effective use of interviews. In the Lincoln FELP 
project, intake interviews are usually conducted at the project site. In 
addition to providing staff important background information about 
the learners, intake sessions familiarize learners with project services 
and enable staff members and learners to get acquainted. 

Lytle (1989) and Auerbach (1990) suggest that, in addition to 
direct questions, interviews may also include opportunities for adult 
learners to explore their abilities to read and write in their native 
language and in English. A reading sample might be used during the 
interview to assess the pers( 'n's reading ability. For example, an adult 
learner could be presented with a variety of reading materials written 
in both languages: newspapers, story books, labels from food, greeting 
cards, rental agreements, letters from school, or prescription labels. 
Learners could be asked to select items they would like to read while 
staff assessed their reading ability. 

To assess writing ability in their native language or English, 
learners might be asked to produce writing samples as shown in 
Example 5. Chapter IV contains alternative instruments and other 
suggestions for assessing reading and writing abilities. 

Intake Assessment: When Time is a Factor 

Because of a shortage of staff and time, many FELP projects 
have combined interviews with p)erformance samples. For instance, an 
intake procedure might begin with an interview in which staff mem- 
bers ask questions in the learner's native language. Next, the learners 
could develop writing samples in response to specific prompts. The 
final part of the assessment could include informal questions and 
conversations between the staff members and adult learners. Project 
staff need to make assessments as non-threatening as possible because 
some adult learners may not have participated extensively in formal 
educational activities. 

Another practical approach to intake assessment is to use some 
time during the first few days of class to obtain placement and other 
background data from the learners (Auerbach, 1990). For example, 
during an introductory lesson, staff could include a writing exercise in 
which learners are given such writing prompts as: **Describe a favorite 
place in your country/' **Describe a favorite food." **Explain why you 
are happy (or not happy) about living in the United States." The 
responses could provide staff with valuable information about the 
learners' previous experiences, current needs, and language abilities. 

A series of at least three writing samples is useful for determin- 
ing where to place learners in a multi-level program. Staff may want to 
ask participants to write about selected topics in their native language 
as well as in English. Writing samples are not only an effective means 
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The most effective alter- 
native assessment instruments 
are those that are custom made 
or adapted to address the spe- 
cific objectives and curriculum of 
the project. 



of placing participants at a particular literacy level; they also provide 
useful initial examples of writing ability. The samples could be placed 
in a portfolio for comparison with examples of learners' work later in 
the project. For learners who have not yet learned to write, staff 
members could transcribe stories that learners tell in their native 
language or English. 

Reading proficiency also can be assessed with alternative 
approaches during regular class activities. FELP staff could provide 
various reading materials in the learners' native language and in 
English such as stories, newspapers, poems, recipes, and personal 
letters. Participants could then choose an item and read it to the class. 
For more in-depth assessment, a staff member could use a similar 
procedure with an individual learner while another staff member 
works on other activities with the rest of the class. The results of such 
assessments are not only useful for program placement; they also can 
be used as baseline data. 

Collecting Baseline Data 

Initial assessment procedures are useful for collecting baseline 
data that establish a starting point regarding a learner's literacy 
development, parenting experience, knowledge of community re- 
sources, self-esteem, confidence, interaction with children, and other 
outcomes that the staff would like to assess. Some FELP projects use 
standardized tests to establish baseline data. If such formal assess- 
ments are used, learners may need a few days to feel comfortable with 
class before they are tested. 

Although standardized tests can be used for part of a project's 
overall evaluation design, alternative approaches are especially useful 
for initial assessment. Surveys, interviews, writing samples, and 
observation measures should be selected early in the project before 
significant learning has occurred* The most effective alternative 
assessment instruments are those that are custom made or adapted to 
address the specific objectives and curriculum of the project. (Project- 
designed instruments are shown in examples 6, 7, and 8.) 

Pianning Initial Assessments 

This section presents a plan of action for helping FELP staff 
design and administer alternative instruments for initial assessment. 
The first step of the plan is for staff members and their evaluator to 
work together to design or adapt instruments and procedures for 
addressing the particular goals and objectives of the project. For this 
step to be successful, staff members should have a clear understanding 
of the project's literacy framework — that is, the philosophy that 
directs the focus of the project and drives it toward the achievement of 
its goals. Without a literacy framework, the evaluation design will lack 
direction, continuity, and meaning. 

As a second step, project staff need to be creative and innovative 
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Staff members and their 
evaluator should work together 
to design or adapt instruments 
and procedures for addressing 
the particular goals and objec> 
tives of the project. 




"Jjll^ Programs should not be 
imposed on adults; instead, they 
need to be shaped and reshaped 
to fit the learners' needs. 



in developing initial assessment strategies. In addition to using assess- 
ment to identify the needs of learners, staff should use initial assess- 
ment to develop project objectives and curriculum content. Assess- 
ment results also can be used by staff as baseline data for comparing 
tiie learners' growth during the project Alternative approaches to 
assessment enable staff to create tiieir own procedures for document- 
ing and representing the positive effects that the project is having in 
meeting tiie learners* needs. Such documentation validates the col- 
laborative efforts of project staff and learners; it also helps secure 
support from other stakeholders for the improvement, continuation, 
and expansion of the project. 

The third step of the plan involves the use of initial assessment 
data by staff members, learners, and project evaluator. Assessment 
instruments and procedures provide information that is too valuable 
and motivating simply to be sent to the project evaluator or filed away 
in an evaluation report. The results of surveys, interviews, observation 
measures, and performance samples should be analyzed and used 
immediately by staff members, learners, and tiie project evaluator to 
determine if the project's goals, objectives, curriculum content, and 
activities reflect the learners' own needs and goals. 

Adult educators, unlike their wunterparts at tiie elementary and 
secondary levels, do not have students who are compelled by law to 
r end classes. Programs should not be imposed on adults; instead, 
Uiey need to be shaped and reshaped to fit tiie learners' nee4s. In 
effective projects, adults do not attend classes because tiiey have to; 
tiiey participate because tiie project meets tiieir needs. By providing a 
match between tiie learners' goals and instructional content, staff 
members can look forward to regular attendance and stable enroll- 
ments. 

Staff members may use the results of initial assessments to 
demonstrate to learners tiiat tiieir responses were used to plan program 
activities. For example, after administering an initial needs assessment 
survey, staff members can explain to learners tiiat tiieir expressed 
areas of interest were used as tiie basis for choosing tiie curriculum 
content of the project. Sharing such information shows participants 
tiiat tiieir feedback had a direct effect on tiie development of tiie 
curriculum content. The results of initial assessment strategies provide 
a basis for collaborating witii adult learners to develop key parts of tiie 
project such as tiie literacy framework. By expressing tiieir literacy 
needs and describing tiieir own goals at tiie initial stages of tiie project, 
learners can monitor their personal growth and take pride in their 
progress. They also can identify and appreciate tiie effects of tiie 
project on their families and community. 

Summary 

Assessment and evaluation are not procedures tagged onto tiie 
responsibilities of project staff members simply to satisfy the interests 
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Assessment and evalua- 
tion are important strategies for 
ensuring the successful imple- 
mentation of a FELP project. 



of a funding agency or other stakeholders. Instead, they are important 
strategies for ensuring the successful implementation of a FELP 
project. Sensitivity to and awareness of the needs, literacy levels, 
goals, and attitudes of students, whether they are adult learners, 
adolescents, or children, are essential if learning and personal growth 
are to occur. Alternative assessment approaches are especially useful 
for enabling staff to stay in touch with, the successes and failures of 
participants and ultimately to make improvements in the project. 

The project evaluator is an important contributor to the assess- 
ment process. The evaluator needs to work closely with project staff to 
develop alternative instruments and procedures for initial assessment. 
Staff will benefit from the assistance of the evaluator in designing 
instruments that are valid, reliable, and linked to project objectives. 
Such instruments promise to generate useful and meaningful data that 
can be analyzed and summarized periodically for learners, staff, and 
other audiences. 

Chapter IV of this document provides a detailed description of 
instruments and procedures for assessing ongoing progress, and 
Chapter V explains how the results of initial and ongoing assessment 
procedures can be collected, analyzed, used, and reported for various 
purposes. Annual evaluation reports that combine the results of formal 
and informal, qualitative and quantitative, and authentic and standard- 
ized measures will provide a comprehensive, credible picture of the 
level of success achieved by ail participants in a literacy project. 
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Example Q 
NEEDS ASSESSMENT SURVEY 



Name 



Address 



Telephone ( ) 



Native Language 



1. 


How many adults live in your apartment? 


1. 


2. 


How many children in your apartment attend Lincoln Unified School District? 






(Identify individual schools by name) 


2. 


3. 


How many adults currently attend GAIN program English classes? 


3. 


4. 


Where do they attend classes? (e.g., Delta College, adult school) 


4. 


5. 


If there are adults living in this apartment who are not attending GAIN English 






classes, would they attend GAIN English classes if they were held in your 






apartment complex? 


c 

5. 


6. 


How many men would go? 


0. 


7. 


How many women would go? 


1 . 


o 

8. 


Would these adults agree to attend class 4 hours a day, 5 days a week? 


Q 
0. 


9. 


Would they need to have someone outside of the family take care oi their 






children during class? 


9. 


10. 


How many children need child care? 


10. 


11. 


Should class be held in the morning or in the afternoon? 


11. 


12. 


Is it a good idea for us to try to get our classes approved for GAIN, or should 






we have our own literacy program? 


12. 


13. 


If we cannot get our English classes approved for GAIN, would you attend 


13. 




anyway? 


14. 


How many hours per day should classes be held? 


14. 


15. 


How many days a week should classes be held? 


15. 


16. 


What would you like to learn? 


16. 




a. English d. How the schools operates 






b. How to get a job e. How to become a citizen 






c. How to be a good parent 




17 


What is your level of English language skills? 






a. Oral skills: (1) Low (2) Medium (3) High 


17a. 




b. Reading and writing skills: (1) Low (2) Medium (3) High 


17b. 


18 


How long have you been in the United States? 


18. 


19. 


How many children in this family are participating in ESEA, Title VII, programs? 


19. 
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PARENT SURVEY 



Name 



Telephone 



Address 



Native language 



Native country 



Can you read in you native language? 



Can you write in your native language? 



Year of arrival in U.S. 



Single mother 



Single father 



Manried 



Name of Children 



Grade 



Name of School 



1.: 



2.i 



3.! 



4.! 



5 J 



8.1 



10. 



1 . Am you presenQy attending a GAIN class? □ Yes □ No Where? 

2. What time is best for you to attend FELP classes? □ morning □ afternoon □ evenings 

3. What days are best for you to attend class? 

□ Monday □ Tuesday □ Wednesday □ Thursday □ Friday 

4. What do you need to help you attend classes? □ Child care □ Translator 

□ Friend to come with you □ Other 

5. What would you like to learn? 

□ Apartment management (how to work with the apartment manager, get something repaired, ask about 
rent, look for an apartment) 

□ Citizenship (how to become a U.S. citizen) 

□ Communication with legal authorities (attorneys, police, legal support, legal aid, neighborhood watch) 

12 Education (registration, how to notify the school of illness, talk to the teacher/principal, visit your 
child's school, understand grades and progress reports) 

□ English-as-a-second-languagc (oral communication, reading, and writing) 

- 1 - 
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Parent Survey (continued) 

n Health (immunizaiions, hygiene, medical and dental check-ups, how to talk to the doctor and dentist, 
fill out medical forms) 

Z! Mathematics (money, banking, nimibers, addition, subtraction, multiplication, division) 

□ Nutrition (common foods in the U.S., healthful eating) 

□ Parenting/child-rearing (developmental stages, suggestions on how to help your child at home, how to 
commimicate effectively with your child) 

□ Safety (emergencies, car seats, fire prevention, water safety, first-aid) 

□ Other 



6. What kind of help do your children need? 



• 
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Example Q 



APPLICATION FOR FELP CLASSES 



Name(Usi) 


(Firsi) 


(Middle) 


Address (Sircei) 




(Ciiy) 


(Suie) (ZIP Code) 


Telephone number ( ) 


Birthdate 


1 

1 L_: Male Fenrjale 


Native country 


Date of arrival in U.S. 



Did you go to school in your country? □ Yes □ No How long? 
Do you read in your native language? □ Limited C Fluent 
Do you write in your native language? "J Limited 12 Fluent 
What other languages do you speak? 



What other languages do you read? 
What other languages do you write? 



Have you taken English classes before? □ Yes G No How long? 

Where? 

Are you: 



□ Married 

□ Single 
Name of husband/wife? 
Number of children 
Health problems 



□ Divorced 

□ Separated 



Ages 



□ Widow 
•Zj Widower 



Social Security number 



Alien registration number 



ERLC 



OfTice use only — Please do not write below this line. 
Date application received: 
Date of test/s: 
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REGISTRATION FORM 



Family name 
Father 
Mother 



Name of Children 


Name of School 


Grade/Room No. 


^.\ 






2. 






3. 






4.. 


1 




5. 






6. 






7. ! 




8. i 




— -| 

1 

9. 1 


1 

I 


10. ! 




Address 


Telephone ( ) 


What country do you come from? 


How long have you been in the U.S.? 


In Solana Beach? 



ERIC 



What level of education did your father complete? 



Your mother? 



Did your children go to school in your country? 



Father's work 



Mother's work 



More than 40 



Are you enrolled in any other ESL class? □ Yes i~ No 

Do you have books at home? □ Yes □ No 

How many books do you have at home? □ 1-10 □ 1 1-20 □ 21-39 
Are they in English or Spanish? □ English □ Spanish [U Both 

Do you have a library card? □ Yes □ No 

Do you have a television in your home? □ Yes □ No 

Do you watch television programs in English or Spanish? □ English □ Spanish 

- 1 - 



Both 



hi 



Registration Form (continued) 

Do you listen to radio prograiris in English or Spanish? □ English □ Spanish □ Both 
Do you read newspapers in English or w^panish? □ English □ Spanish □ Both 
Do you correspond with your family in Mexico? □ Yes □ No 
Docs your father: 

Understand spoken English? □ Yes C No 

Speak English? □ Yes □ No 

Read English? □ Yes □ No 

Write English? □ Yes □ No 

Read Spanish? □ Yes □ No 

Write Spanish? □ Yes □ No 
Does your mother: 

Understand spoken English? □ Yes [Z No 

Speak English? □ Yes □ No 

Read English? □ Yes □ No 

Write English? □ Yes □ No 

Read Spanish? □ Yes □ No 

Write Spanish? □ Yes □ No 
Do the children: 

Understand spoken English? □ Yes □ No 

Speak English? □ Yes □ No 

Read English? □ Yes □ No 

Write English? □ Yes □ No 

Read Spanish? □ Yes □ No 

Write Spanish? □ Yes G No 
What problems have you encountered in the United States that you would like to discuss in class? 
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Example 0 
INTAKE ASSESSMENT FORM 



Name (Ust) 



(First) 



(Age) 



Write numbers 1-27. 



27 



Write the alphabet. 

A 



Read the sentences and circle the correct word. 



1. He go at went toworic. 



2. She j am i^^ are | thirty-five 



3. How old 



am IS are you? 



4. I I go going went | to buy food yesterday. 



5. We came come came from to the United States five years ago 



6. They has have got five children. 



7. I I has had have been | studying English for six months. 
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Example Q 
INITIAL ASSESSMENT FORM 



Name 



Date 



How many children do you have? 



Name of Children 



Name of School 



1/ 



2.1 



3 ' 



4: 



5 J 



6J 



8.; 



9.' 



Grade 



10. 



Please mark the appropriate answer: 

1. How many hours per week have you spent helping your 

child (children) with homework? 1 or less 

2. How many times have you met with the teacher or other 

school personnel? 1 or less 

3. How many times have you attended PTA meetings? 1 or less 

4. How many times have you attended meetings of the 

Bilingual Advisory Committee? 1 or less 

5. How many times have you attended sports activities 

offered by the school? 1 or less 

6. How many times have you participated in some school 

activity such as a trip to a museum? 1 or less 

7. How many times have you done volunteer work at the 

school? 1 or less 

8. How many times have you participated in other school 

activities such as conferences for parents? 1 or less 



2 3 4 or more 

2 3 4 or more 

2 3 4 or more 

2 3 4 or more 

2 3 4 or more 

2 3 4 or more 

2 3 4 or more 

2 3 4 or more 
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PRE/POST TEST: COOKING AND NUTRITION 



Circle the foods that are good snacks. 




candy apple doughnut peanuts soda 




carrots orange milk potato chips cookies 

Circle the correct answer. 



1, What's this? 


2. What's this? 


3. What's this? 


4. What's this? 










doughnut 


cake 


cheese 


hamburger 


pie 


cookies 


sandwich 


salad 


hamburger 


hot dog 


meatloaf 


pizza 



Make an X on the foods which are not good snacks. 




Adapted from Holt. Grace D. Parenting Curriculum for Language Minority Parents. Sacramento: Cross Cuitura! Resource Center. California 
State University, 1988 
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PRE/POST TEST: THE U.S. SCHOOL SYSTEM 



Circle the one correct answer for the questions below. 

1. The report a school sends home to tell about a student's progress in school is called a 

A. Cinch notice 

B. Emergency card 

C. Report cani 

D. Absence report 

2. The top person in an elementary school is called a 

A. Librarian 

B. Principal 

C. Teacher 

D. Vice-principal 

3. High school starts from grade 

A. Six 

B. One 

C. Nine 

D. Twelve 

4. A student checks out books from the 

A Cafeteria 

B. Playground 

C. Office 

D. Library 

5. When a child is too sick to go to school, he or she should 

A. Stay home and play outside 

B. Rest inside the house 

C. Watch TV all day 

D. None of these 

6. What things are needed for a child when registering for school? 

A. Birth certificate 

B. Shot records 

C. Physical examination 

D. All of the above 

Answer the following questions true (yes) or false (no). 

7. Children need more sleep than adults. 

8. A parent cannot help a child in school if the parent does not speak English. 

9. Schools do not want the parents to help their children with school work. 

10. Teenagers go to elementary school. 



1 1 . Have you visited a public school since your arrival in the United Stales? 17 Yes ZI; No 

- 1 - 
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Pre/Post Test: The American School System (continued) 

If you have a child in school, answer the following questions. 

12. What grades are your children in? 



13. What schools do your children attend? 



14. Where are your children's schools located? 



15. What are the names of your children's teachers? 
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FELP Resource Guide 



SURVEY OF USAGE 

Chapter 1(1. Initial Assessment: First Step to Success 

This resource guide must be helpful to you, the FELP director or evaluator, in order to be considered 
a success. In accepting the field test edition of the guide, we hope that you agree to participate in this 
usage survey. We plan to revise the guide based on your comments and criticisms; it then will be 
published for a broader audience. Please share with us your critical comments. After reviewing the 
chapter, use the scale below to indicate whether the guide is written clearly; the scale indicates that 
it is 

1. Clearly written and understandable, the format is clean; 

2. Generally clear, but some wordings or format could be Improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at all clear, not at all understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. In each of the same areas, please use the scale to indicate that the GUIDE has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal; or 

4. Information that I don't need and won't use. 



Rate clarity 
12 3 4 
12 3 4 
Not applicable 

12 3 4 
12 3 4 
12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 



Area to be rated 

Format of the chapter 

Wording of the chapter 

The general ideas and information presented 

The role of assessment: Needs assessment survey 

Related needs assessment activities 
Expanded view of literacy 



initial assessment: 



Rate information 
Not applicable 
Not applicable 
12 3 4 



Dynamic continuum - Parent survey 
Intake assessment - Personal interview 
intake assessment - Performance samples 
Collecting baseline data 



Planning initial assessments - Designing alternative instruments 



1 



2 3 4 
2 3 4 
2 3 4 



2 
2 
2 
2 



12 3 4 



Thank you for your help. If there are other areas that you feel should be included in this chapter, or 
other comments about the chapter that you would like to make, please feel free to do so. You do not 



need to provide your name. 
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«3* Please return your completed survey. You may fold, staple, and stamp this page - the address is on the back 
side, if you prefer, you may FAX it to Daniel D. Holt, Bilinguol Education Office at (9 16} 657-2928. We also 
would appreciate receiving a copy of any alternative assessments you create as a result of this guide. 

i o 



Fold here 




Daniel D. Holt 

Bilingual Education Office, California Department of Education 

PO Box 944272 
Sacramento, CA 94244-2720 



Fold here 



QSSESSING QUCCESS 



9 



ERIC 



CHAPTER IV 

ASSESSING 



PROGRESS: 



PROGRESSING? 




Heide Spruck Wrigiey 
Aguirre International 
San Mateo, California 



To judge how well a project is working, staff members need to take the 
educational pulse of the program periodically and assess learners' progress. 
Progress assessment can help staff determine the extent to which their 
instructional efforts are leading to beneficial results and having positive 
effects on the learners* lives. The assessments also give participants a ',l-;sc 
of accomplishment by showing them that they are making headway toward 
attaining their goals. To be effective, progress assessments need to include 
several components: 

1. Baseline data* that show what participants presently know; 

2. Indicators of progress or descriptors of what counts as success; 

3. Methods for gathering information on a continuous basis; and 

4. Plans for synthesizing and analyzing the data that are collected. 



♦ Sec the Glossary on pages 114-115 of this publication for definitions of importani terms 
that appear in this chapter. 
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This chapter shows how staff can develop progress indicators 
based on the objectives of their project (see chapters I and II for more 
on objectives). Although the chapter focuses on projects funded under 
ESEA, Tide ^ ' \ Family English Literacy Programs (FELP), the 
information could be adapted for other literacy projects for language- 
minority adults. 

Six sets of instructional activities are presented, each of which 
focuses on particular learner outcomes and provides examples of 
alternative approaches for assessing the learners' progress toward the 
outcomes. The sections are titled (1) Helping Learners and Staff Get 
Acquainted; (2) Linking Project Activities with Family Life; (3) 
Learning to Read to Operate Machines; (4) Learning to Write Direc- 
tions; (5) Biiilding Self-Esteem and Taking Literacy Beyond the 
Classroom Walls; and (6) Developing Literacy Through Telling and 
Writing Stories. 

The chapter illustrates how learners' performances in these 
activities can be assessed with observation measures, interviews, 
questionnaires, and performance samples. The examples of alternative 
assessment instruments presented in this chapter are not intended as 
ready-to-use assessment tools. FELP staff should adapt the instru- 
ments and procedures based on the unique characteristics and objec- 
tives of their project. Before presenting the activities and instruments, 
a rationale will be provided for using alternative approaches for 
assessing learners' ongoing progress. 



Alternative assessments 
can focus on the way learners 
use language and literacy as 
opposed to how much they know 
about grammar, phonics, or 
spelling. 



Characteristics of Alternative Approaches 
to Assessing Ongoing Progress 

Conventional assessments often use pre- and post-tests to com- 
pare how much learners know before and after an instructional unit. 
Alternative assessments can go one step further: rather than focusing 
only on knowledge and skills, they seek to show changes in learners' 
attitudes, social interactions, and communication patterns that may be 
attributed to their participation in the program. In contrast to many 
standardized assessments, alternative approaches can link instructional 
content with assessment. In addition, alternative assessments are o.^en 
participatory, involving learners as partners in the assessment process. 

Linking Curriculum with Alternative Assessment 

Alternative assessments can focus on the way learners use 
language and literacy as opposed to how much they know about 
grammar, phonics, or spelling. For example, a traditional test might 
assess whether students know irregular verb tenses, can find words 
that rhyme with cat, label the parts of the body, or answer comprehen- 
sion questions about a paragraph. An alternative assessment, on the 
other hand, might seek to examine how weU a learner can explain a 
problem, read a report card, or write a note to a neighbor. Alternative 
assessments also can be used to report on the role that literacy plays in 
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During the eariy stages of 
a project, it is important for 
learners to have opportunities to 
discuss the role that literacy 
plays in their lives. 



learners' lives and to document the life changes that take place as 
learners become involved in family literacy. 

Alternative assessments are particularly useful with a language 
and literacy . curriculum that emphasizes meaning over form and 
communication over structure. Such assessments are also effective for 
learner-centered approaches to teaching English-as-a-second-language 
(ESL) literacy. One recent research study on successful programs and 
promising practices defined a learner-centered approach to ESL 
literacy as **supporting language-minority adults with little English 
and little formal education in their efforts to understand and use 
English in its many forms (oral and written, including prose, docu- 
ment, and quantitative literacy), in a variety of contexts (family, 
community, school, work), so that they can reach their fullest potential 
and achieve their own goals (personal, professional, academic)" 
(Wrigley and Gulh, 1992, p. 7). 

Because a learner-centered curriculum tries to be responsive to 
the participants' changing needs and goals, learner-centered assess- 
ment also needs to be dynamic and flexible. Such assessment should 
be capable of capturing incidental learning and unforeseen problems 
and of demonstrating the extent to which program objectives have 
been met. A collaborative effort among teaching staff, learners, and 
those evaluating the project helps to ensure effective planning and 
implementation of alternative assessment (see Chapter II for a detailed 
description of the planning process). 

The next section is the first of six sets of instructional activities 
that provide a context for illustrating a variety of alternative ap- 
proaches to assessing ongoing progress. Linking instructional activi- 
ties with aUemative assessment provides a multi-faceted view of the 
learners' involvement and progress in a project. 

Helping Learners and Staff Get Acquainted 

During the early stages of a project, it is important for learners to 
have opportunities to discuss the role that literacy plays in their lives. 
Through a series of exploratory activities, learners may identify their 
goals for literacy, describe the way they use literacy at home and in 
the community, explain their hopes and dreams for their children, and 
talk about their interests in acquiring literacy in their native language 
and their goals for learning English. This section presents activities 
and assessment procedures that can be used at the beginning of a 
project to obtain information about the learners' needs that might be 
addressed in the program. (Chapter III also provides initial assessment 
strategies for obtaining information for developing instructional 
objectives and curriculum content.) 

Surveys and questionnaires can be usea jUSt before classes begin 
to assess the learners' needs. After instruction is underway, additional 
information can be gathered effectively in small group activities that 
are implemented during the first two weeks and recycled in various 
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To develop useful learner- 
centered assessments, staff 
members need to gain and main- 
tain the trust of the learners so 
that they feel free to share their 
experiences, ideas, and aspira- 
tions with their peers. 



ways throughout the remainder of the project. To develop useful 
learner-centered assessments, staff members need to gain and maintain 
the trust of the learners so that they feel free to share their experiences, 
ideas, and aspirations with their peers. 

Small group and pair activities are especially helpful in allowing 
staff to (1) note how learners interact with each other; (2) observe the 
learning strategies they use; (3) gain insights into the learners' com- 
munication skills and literacy practices (in English and in their native 
language); and (4) recognize the skills and abilities that learners bring 
to class. 

Here's What to Do 

To find out more about the learners and help them get to know 
one another, staff members can use (1) grids in which participants 
match information about a classmate with the person's name: for 
example, "Who has two children and comes from Vietnam?'*; (2) 
maps and pictures that allow learners to talk about their na'jve coun- 
tries; (3) family photographs brought from home; and (4) language 
experience stories in which learners tell about a signif ^ant event such 
as, "My First Day in the United States." 

Learners also can get acquainted by intervie^/ing each other. 
Gillespie (1990) suggests that teachers cut pictu' es in half; ask learners 
to choose half a picture out of a box; find the person who has the other 
half; and interview that person. Interview topics could include ques- 
tions about the learners' 

1 . Interests, likes, and dislikes; 

2. Background knowledge, skills, and occupation; 

3. Reasons for enrolling in the FELP project; 

4. Family life and parent-child activities; and 

5. Experiences witli schooling in their native country and the United 
States. 

Follow-up activities can address such questions as, "When and 
where do you use English and why?" "When do you use your native 
language?" "What are your educational goals for yourself and for your 
children?" Learners also can be asked about the role of literacy 
brokers in the community — individuals who translate for others and 
help them negotiate paths through the maze of U.S. bureaucracies and 
otiier obstacles (Weinstein-Shr, 1990). Of particular interest for family 
literacy might be the role that children play in mediating language for 
their parents and grandparents. FELP staff can use such investigations 
io create in-class discussions about cross-cultural differences in 
communication, schooling, or home health care. These discussions 
could then lead to instructional activities that provide opportunities for 
sharing feelings and ideas and for acquiring other knowledge and 
information. 
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Ij^^ Collaborations among 
participants help to create a com- 
munity of learners and allow staff 
to assess the learners aJong vari- 
ous dimensions of literacy. 



Assessment Strategies 

Activities in which participants interact, leam together, and teach 
each other provide staff with opportunities to assess what learners 
know, what role literacy plays in their lives, when and where tliey use 
English and their native la-iguage, and what educational values they 
hold. Collaborations among participants help to create a community of 
learners and allow staff to assess the learners along various dimen- 
sions of literacy such as: 

1. Levels of self-esteem and confidence in interacting with others; 

2. Ability to understand spoken and written English; 

3. Ability to express ideas orally and in writing; 

4. Patterns of native language use at home and in the community; 

5. Knowledge about cross-cultural differences in institutions, 
services, and life-skill areas; and 

6. Levels of independence and degi^ee of transfer of skills and 
knowledge beyond the classroom, including changes in behavior 
desired by participants. 

Staff members who are uncertain about which of these dimen- 
sions to focus on initially can design an observation measure for each 
learner that captures infonnation in each of the areas until categories 
worth pursuing begin to emerge. Or staff can create a more simplified 
version that would focus on fewer areas. Example 1 is an observation 
measure that can be used for obtaining basic infonnation about the 
learners. 

Information from the observation measure in Example 1 can be 
used to: 

L Generate baseline data for monitoring learners' progress over 
time; 

2. Provide infonnation for staff members to share regarding indi- 
vidual learners and to detect common patterns in the learners' 
responses to classroom activities; and 

3. Create classroom activities that address learners* needs. 

Staff members can use questions such as the following to guide 
their discussion and to develop descriptors for various levels of 
progress: 

1 . What differences do we see between more advanced and lower- 
level learners in reading, writing, speaking, and listening abili- 
ties? 

2. What evidence of coping skills do we observe? What do we 
know about the strateg'* that learners use to solve problems and 
communicate in English even though they have only minimal 
English and are just barely literate? 

3. How much do individual learners already know about laws, 
regulations, and institutions that affect their community (for 
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Example © 
Learner Profile/Observation Card 



Type of assessment: Observation measure 

Purpose: To record information about the knowledge, skills, and strategies that learners possess and 
the level of confidence they display in interacting with each other and to collect anecdotal evidence 
that shows the extent to which participants are using the information gained in class in their lives 
outside the classroom. 

Method: Open-ended charts allow teachers to take notes on what leamers do and say. 



Learner Profile/Observation Card 



Date 


Teacher 


Unit 


Class 


Learner's name 


Learner goals and interests: 

• Communication skills 

listening and reading 
speaking and writing 




• Coping strategies and problem solving 

linguistic 
non-verbal/social 




• Content know^ledge 

institutions 
services 
life skills 




• Levels of confidence 
self 

one-on-one 
small group 
large group 





Transfer of skills and knowledge beyond the classroom: 
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By developing descriptors 
of learners' behavior at various 
stages of development, staff are 
able to identify changes in learn- 
ers' performance over the course 
of the project. 



Initial program activities 
need to encourage participants to 
leam from and teach each other by 
observingand doing. 



example, schools, health care providers, and social service 
agencies)? 

4. What evidence do we have ihal learners are applying in their 
personal lives what we are teaching in the project? 

5. What are the characteristics displayed by independent learners? 
How are such learners different from those who are overly 
dependent on their peers or the teacher? 

By developing descriptors of learners' behavior at various stages 
of development, staff are able to identify changes in learners' perfor- 
mance over the course of the project. Staff members can continue to 
aJd meaningful information to learner profiles as the project is imple- 
mented. When assessment and instructional activities work in tandem, 
learners' needs and their progress are illuminated, enabling staff to 
gain clearer insights into how the project can be tailored more appro- 
priately to the learners. 

Linking Project Activities with Family Life 

This section shows examples of how instructional activities, goal 
setting, and assessment can be linked with family life. First, learners 
work together to create some kind of joint project. Next, parents talk 
about the kinds of activities they do with their children at home and 
discuss how these activities can support children in their own learning. 
Parents then decide what they would like to do with their children 
during the next class sessions. Finally, parents talk about the time they 
spent with their children and what their experiences have meant to 
them. 

Here's What to Do 

Initial program activities need to encourage participants to leam 
from and teach each other by observing and doing. For example, staff 
members, parents, and children can work and play together to create 
collages, holiday greetings, announcements, or birthday cards. Staff 
might start by showing some interesting examples and by demonstrat- 
ing a particular technique for creating the materials. While they work 
with paper arxl scissors, participants leam the English names for 
colors, shapes, and actions such as cutting, folding, and tearing. 

In subsequent activities, participants share information, describe 
their materials, or teach each other a particular technique. Similar 
activities can include making paper airplanes, origami (paper folding), 
or party decorations. (For an example of a lesson on making and 
playing traditional instruments, see Quintero and Huerta-Macias, 
1990.) Next, parents can discuss some of the activities they do with 
their children at home and talk about the kinds of reading and writing 
they do in tneir native language or in English. 
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Activities that involve 
parents and children working 
together offer staff a starting 
point for discussing the social 
interactions among family mem- 
bers and to understand the 
literacy practices used in 
the home. 



till Questionnaires are useful 

for finding out about literacy 
practices that parents use at 
home. 



Assessment Strategies 

Activities that involve parents and children woridng together 
offer staff a starting point for discussing the social interactions among 
family members and to understand the literacy practices used in the 
home. To learn more about the participants' families, staff can develop 
questionnaires in which parents, working individually, in pairs, or in 
groups, list the activities they do at home with their children. 

E ' mple 2 is an illustration of a questionnaire for generating a 
list of activities that parents and children do together. The list gener- 
ated by the learners can become baseline data for documenting 
changes over time in parent-child interactions (see Example 3). The 
assessments shown in examples 2 and 3 could be conducted in the 
learners* native language or in English. 

In ensuing activities, staff members can provide information to 
parents about learning strategies used by children and present ex- 
amples of how parents can support their children's learning at home. 
For example, parents lct j've children opportunities to observe others, 
learn new things with siblings, w ,k questions, or discuss cause and 
effect relationships. Using a questionnaire like the one shown in 
Example 4, parents can discuss activities they consider useful for 
helping their children. 

Next, parents can identify the kinds of activities they plan to 
undertake with their children and the frequency with which they might 
conduct each activity (see E;^^ple 5). These activities could become 
part of an individualizedLp^^n that helps leamers document progress 
toward meeting their pwn goais and the objectives of the project. 
Parents could use a questionnaire such as Example 6 for reporting the 
activities they try with their children and for determining to what 
extent they have met the goals in their plan. 

Examples 4-6 can be adapted to meet the specific purposes of a 
FELP project. For example, the complexity of the English vocabulary 
and structures used in the instruments could be tailored to the leamers* 
ability. In addition, the instruments could be translated and adminis- 
tered in the leamers* native language. Some projects also might have 
staff members or leamers use the instruments for individual interviews 
in English or the native language. 



Learning to Read to Operate Machines 

The activities in this section are designed to help participants 
develop their receptive skills of listening and reading. The focus of the 
activities is on introducing beginning learners to reading through 
environmental print (signs and symbols in the community) and im- 
perative print (messages that tell citizens what to do and not to do). An 
observation measure is presented for assessing listening and reading 
abilities. 
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Example Q 

Questionnaire 



Type of assessment: Self-assessment; questionnaire 

Purpose: To identify literacy practices in the home and record parent-child interactions. Information 
provides baseline data for documenting changes over time. 

Method: A list of activities is generated as part of whole-group or small-group discussions or after 
learners interview each other. If learners are not proficient in English, the activity can take place 
in their native language. In cases where the learners are not Hterate, the teacher or a literate 
partner can record responses. 

Prompt to learners: Parents and children can do many things together. They go to the park on 
Sunday, go fishing, cook food, clean the house, go shopping, watch TV, or look at magazines. In 
many families, parents help their children with homework or check their assignments. What do 
you do together with members of your family? 
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Example © 

Self-assessment Instrument 



Type of assessment: Self-assessment of literacy activities by freqency count 

Purpose: To identify family activities that learners engage in at home. The instrument can be used for 
generating baselme data and documenting changes during the implementation of the project. 

Prompt to learners: You do many things together as a family. TeU us how often you do these activities 
(Be sure to bst any activities you do that are not on the list.) 



Family activity 


Almost 
every day 


Once or 
twice a week 


Several 
times a month 


Once or | 
twice a month \ Seldom 


Go to the par1< 








1 
i 
1 


Watch TV together 










Cook or clean house together 








• 


Fix things that are broken 








i 


Go fishing | 






I 


j 

Do homework together I 








! 


Look at a book or magazine 








i 
1 
1 



Play games 



Ottier activities \ 

\ 
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Questionnaire 

Type of assessment: Self-assessment; open-ended questionnaire 

Purpose: To record home literacy events and activities that parents regard as essential and to gain 
insights into educational values and opinions about learning. 

Method: As part of whole- or small-group discussions or through pair activities, learners discuss their 
views of how children learn and how parents can help their children learn. 

In my opinion, these activities help children learn: 

Talking about (e.g.. field trips and birthday parties) 



Teaching children to (e.g., ride a bicycle and bake cookies) 



Helping children with (e.g., writing a letter and doing math problems) 



Asking children questions about (e.g., their friends, favorite teacher, and personal problems) 



Telling children that (e.g., everyone makes mistakes and doing homework is important) 
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Self-assessment instrument 



Type of assessment: Self-assessment; frequency count 

Purpose: To set personal goals and develop an individualized plan for documenting progress. 

Prompt to learners: Based on the home literacy activities identified in the questionnaire (Example 4), list 
activities that you plan to undertake with your children and indicate how often you might do each. 





j Almost 


Once or 


1 1 
Several i Once or | 


Family activity 


every day 


twice a week 


times a month ! twice a month \ Seldom 


i 

My children and I plan to . . . ' 

1 

. 


\ -[ 
1 1 
i 1 

1 • 



go to the library 



1 
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Example O 



Questionnaire 



Type of assessment: Self-assessment; open-ended questionnaire 

Purpose: To record literacy events and learner's accomplishments related to literacy and to compare 
goals in the learner's individualized plan with actual activities. 

Method: Learners talk about the kinds of activities they did together with their children and discuss the 
literacy events that have taken place. The learner or a literate partner records responses. At key 
points in the cycle the list is compared with the learner's individualized plan. The plan is then 
discussed anew and goals are renegotiated based on the learner's current perspectives. 



Learner's name 



Date 



This week, I helped my son with his math. 
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''I'Jl^ With practice, learners 
can develop their own under- 
standing of the meanings of the 
descriptors and begin to evaluate 
their own performance. At more 
advanced stages, they can evalu- 
ate their peers and give construc- 
tive feedback about their perfor- 
mance. 



Field trips can be used to give adult learners opportunities to 
operate a variety of machines such as those for making photocopies or 
obtaining soft drinks and change. Participants also can learn how to 
use a public telephone or operate an automatic teller machine. 

Working in pairs or small groups, learners discuss the instruc- 
tions for the machines and practice what is required to operate them. 
Learners might use a polaroid camera for taking pictures of instruc- 
tions that they find hard to understand. During a field trip, they also 
can write down examples of imperative print, such as Exit, Stop, and 
No Turn on Led. Staff members could select several sets of the 
instructions for additional practice in class. 

During classroom activities, learners can identify the machines 
that they found particularly useful, interesting, or difficult to operate. 
They might decide which instructions they would like to study further 
in literacy activities such as reading simple imperatives, matching 
instructions with appropriate machines, or putting instructions in 
sequence. For example, the sequence of instructions for using a public 
telephone phone would be: (1) Lift receiver, (2) Listen for the dial 
tone; (3) Deposit 20 cents; and (4) Dial the number. 

Follow up: Learners can develop the language abilities related to 
operating other machines such as an audio tape recorder, video cas- 
sette recorder, stapler, three-hole punch, or computer. As learners 
become more proficient, they might woric with longer and more 
complex instructions for such tasks as assembling simple toys, making 
crafts, following recipes, or completing job application forms. 

Assessment Strategies 

Activities on how things woric give staff the opportunity to assess 
the learners* reading and listening skills. To generalize from the 
assessment data, similar activities should be repeated throughout the 
instructional cycle. Instructions can be given orally to test listening 
comprehension or in writing to assess reading skills. In both cases, the 
instructions used for assessment should be somewhat familiar to 
learners so that they do not become overwhelmed or discouraged by 
the task. Example 7 is an illustration of an observation measure for 
assessing listening and reading abilities. 

To develop a consensus on the meanings of the descriptors 
C*Wilh ease,'' "With some support,'' and so forth) and establish the 
reliability of the observation measure, staff members should use the 
instrument with similar adult learners under a variety of circum- 
stances. With practice, learners can develop their own understanding 
of the meanings of the descriptors and begin to evaluate their own 
^performance. At more advanced stages, they can evaluate their peers 
and give constructive feedback about their performance. 



ERIC 



Example Q 



Observation Measure for 
Listening and Reading 



Type of assessment: Observation measure 
Purpose: To assess listening and reading skills 

Prompt to students: As learners listen to or read instructions, staff members assess their performance and 
check the appropriate box in the chart 

Method: After learners listen to or read instructions, staff members assess their performance and check 
the appropriate box in the chart. 

For example: After learning how to make instant chocolate pudding for a class party, learners arc asked 
to follow oral or written instructions for making vanilla pudding (familiar instructions) and gelatin 
dessert (new instructions). 

Learners who can comprehend and read instructions without asking questions or looking terribly 
puzzled are rated "With ease.** Those who occasionally ask to have a word provided, a sentence 
modeled, or seek help from peers are rated "With some support." Learners who need to consult a 
dictionary, ask peers for translation, or constantly observe others are checked "With great difficulty.*' 





Receptive skills 


With ease 


With some 
support 


With great 
difficuity 


' Not at 
all 


Listening 


Follows instructions 






i 




Fellows newinstaictions 


! 


; 
; 

■ 


Reading 


Reads fam/7/ar instructions 

' i 


! 

1 

I 



Reads new instructions 
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Learning to Write Directions 



In cooperative activities, 
participants realize that each 
member of the group can be a 
teacher and a learner. 



This section presents cooperative activities in which participants 
teach and learn from each other by talking about a particular skill or 
ability that they possess. The activities work well in a multi-level class 
because they give less proficient learners a chance to develop knowl- 
edge by observing and responding non-verbally while more advanced 
learners enjoy the special chaUenge of teaching others. An observation 
measure is presented for assessing learners' speaking and writing 
abilities. 

Here's What to Do 

In cooperative activities, participants realize that each member of 
the group can be a teacher and a learner. To introduce the activity, 
staff members can ask parents what they teach their children and what 
their children teach them. It may be interesting for adult learners to 
talk about what they learned from their parents. Learners might also 
discuss how they help their neighbors and friends. These activities 
allow staff members to identify skills that participants have, such as 
repairing a flat tire, preparing a home remedy for a sick child, fixing a 
broken window, getting rid of stains, or braiding long hair. 

During these activities, staff members can validate knowledge 
that learners possess such as knowing how to find a bilingual doctor; 
where to get free legal advice; how to talk to a social worker, and 
where to find a video store that stocks films in the learners' native 
language. Staff members might make a list of skills represented in the 
group and ask learners to decide which ones they would like to 
develop further. Staff can also discuss the possibility of bringing in 
outside speakers from legal aid and public health agencies, the Red 
Cross, or the community relations uviu of the police department. 
Presenters can talk about topic? suc'n as fighting an eviction notice, 
practicing safe sex, giving blood, or oroiecting a home against burglar- 
ies. 

Staff members can risk panicipants to choose a particular task 
that intenests ihein sjch as moiling tamales or a fruit salad; sorting 
coupons to use the gro'^^f^ry store; demonstrating first-aid; playing 
the kazco: making pop^.orr putting up a bulletin board; or taking 
polaroid pictures. Gi "nrp members might work together to write 
instructions for others to cairy out. After discussing and demonstrating 
the instmctions in small groups, indivJdi^ai learners can demonstrate 
tiie task to the class usmg both verbal and non-verbal cues to illustrate 
their points. To evaluate the learners' speaking and writing abilities 
related to using instructions, staff members can use an observation 
measure similar to the one shown in Example 8. 

Assessment Strategies 

Tht observation measure shown in Example 8 demonstrates how 
an instnjment can be developed for specific purposes in a project. In 
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Example O 



Observation Measure for 
Speaking and Writing 



Type of assessment: Observation measure 
Purpose: To assess speaking and writing skills 

Method: As learners talk about or write down instructions, staff members assess their performance and 
check the appropriate box in the chart. 

For example: After leaming how to make buttered toast, learners are asked to follow oral or written 
instructions for making toast with peanut butter and jelly (familiar instructions) and bagels and cream 
cheese (new instructions). 

Learners who can produce and write instructions without asking questions or looking terribly puzzled 
are rated **With ease." Those who occasionally ask to have a word provided, a sentence modeled, or 
seek help from peers are rated "With some support." Learners who need to consult a dictionary, ask 
peers for translation, or constantly observe others are checked "With great difficulty." 





Productive skills 


With ease 


With some 
support 


With great 
difficulty 


Not at 
all 


Speaking 


Produces familiar spoken 
instructions that are 
clear and comprehensible 










Produces new spoken 
instructions that are clear 
and comprehensible 









1 


Writing 


Produces familiar v^rWien 
instojctions that are clear 
and comprehensible 










Produces new written 
instOJCtions that are clear 
and comprehensible 
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preparation for a party 
or a holiday, learners might work 
together to prepare a meal, pro- 
vide entertainment, write invita- 
tions, and make decorations for 
tiie room. 



Research has shown that 
learners who interact with and 
use others as resources make 
significantly greater gains in 
learning than do students who 
work alone. 



this case, the instrument was designed to measure learners' progress in 
improving their speaking and writing abilities related to using instruc- 
tions for particular tasks. 

Another assessment strategy would be to put each learner's 
written instructions in a portfolio for comparison with future writing 
samples. Copies of the instructions could be given to other learners to 
try to follow. Staff might use selected sets of instructions for subse- 
quent literacy activities such as doing fill-in-the-blank or cloze exer- 
cises, completing sentences that require longer phrases, unscrambling 
words in sentences, sequencing instructions, and matching instructional 
steps with pictures. 

For intermediate leamers, the types of activities described here 
can be extended to teach functional literacy skills such as filling out 
school registration forms, understanding immunization records, and 
telephoning the school for information. (For other examples of literacy 
activities for intenmediate students, see Holt's Parenting Curriculum 
for Language Minority Parents: Teacher's Activities Guide, 1988.) 

Staff members also can design writing activities for the whole 
class. For example, in preparation for a party or a holiday, leamers 
might work together to prepare a meal, provide entertainment, write 
invitations, and make decorations for the room. The activities could 
then be written up, either as work plans, narratives, or items for a class 
newsletter. The written work could become the basis for further 
language and literacy development. 

Building SeifEsteem and Taking Literacy 
Beyond the Classroom Walls 

The alternative assessment instruments discussed tlius far focus 
on language use; that is, they seek to assess how well leamers can 
perform specific language-related tasks. Although these instruments 
can document changes in language abilities, they do not provide 
information on affective factors such as self-confidence, self-esteem, or 
motivation. Neither do the instruments provide data that would indicate 
how well leamers work together or to what extent they transfer knowl- 
edge gained in the classroom to their lives beyond the school. 

Affective factors deserve the attention of FELP project staff 
because of the importance of self-confidence and self-esteem in 
influencing not only the learners' language development but also their 
general level of success in adapting to the cultural challenges they face 
at home, in the community, and at work. Research has shown that 
leamers who interact with and use others as resources make signifi- 
cantly greater gains in Icaming than do students who work alone 
(Kagan, 1986; McGroarty, in press). Similarly, studies in cognition 
demonstrate that language and literacy development is increased if 
leamers can link classroom knowledge to real-life contexts (Anderson 
an/i ^-a-T^on, 1988: Anderson, 1988; Smith, 1975). 
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Here's What to Do 



Project staff can use a variety of activities for helping learners 
develop self-esteem and improve their interaction with others. Activi- 
ties should enable learners to demonstrate the knowledge, insights, and 
abilities they have developed in responding to the real-life challenges 
they have faced. The following are activities that FELP staff might try 
in their project, or they may want to design their owa 

1. Case studies in which learners are presented with a familiar 
problem and asked to brainstorm possible solutions. For example, 
a case might involve a daughter who wants to play on the volley- 
ball team, but the mother, a single parent, is worried about her 
daughter's being av/ay from home for practice and out-of-tovm 
games, 

2. Situations that illustrate a significant economic, social, or cultural 
issue, such as lack of affordable housing, street crime, and racial 
discrimination. The situations, sometimes called codes^ can be 
represented through short readings, dialogues, and pictures. They 
can provide starting points for the learners to discuss their con- 
cerns and to brainstorm solutions. Codes are based on problem- 
posing pedagogy developed by Paulo Freire (see Wallerstein 
[1983] for similar activities). 

3. Simulations and role plays in which learners play themselves in 
situations such as shopping at a store, participating in a parent- 
teacher conference, or dealing with uncooperative landlords; 

4. Projects such as developing a class biography, producing a 
resource guide for newcomers to the community, setting up a flea 
market or crafts fair at the school, or producing a puppet show for 
the children; 

5. Lists and charts of individual accomplishments that are posted on 
classroom walls. Learners use their own words to describe 
achievements related to language, literacy, and learning that they 
are proud of. 

6. Language experience stories in which learners tell of the struggles 
in their lives and celebrate their successes in becoming biliterate 
and bicultural (for examples of learner-generated stories, see 
Weinstein-Shr, 1992). 

Assessment Strategies 

The observation measure shov^m in Example 9 can be used to (1) 
record changes in the learners* confidence; (2) document levels of 
participation in pair or group activities; and (3) show transfer of skills 
to actual life experiences. By recording infomation about the three 
affective areas in one chart, staff members can analyze relationships 
among the domains. For example, the data might indicate that as an 
individual learner becomes more confident, he/she participates more 
freely and transfers the knowledge gained in class more readily to real 



ERLC 



^ ^ A$$§s$ing SuccM 



life. Results of the measure might also illustrate that levels of self- 
confidence are related to levels of participation, as in the case of 
learners who collaborate more freely as they become more self- 
confident. When interpreting the results, staff members need to be 
aware that factors outside the FELP project also may influence the 
learners* level of self-confidence. 



When interpreting the 
results, staff members need to be 
aware that factors outside the 
FELP FKoject also may influence 
the learners' level of self-confi- 
dence. 



Example O 

Type of assessnrv^nt: Observation measure 

Purpose: To assess learners* p)erformance in non-linguistic 
domains related to learning and literacy development. The instru- 
ment can be used for generating baseline data for documenting 
changes during the implementation of the project. 

Method: Staff members write COTiments in the appropriate 
columns as they listen to learners talk about their abilities and 
work together in pairs or in small groups. Anecdotes about 
learners' lives outside the classroom are also used as sources of 
assessment data. 

The following are detailed descriptors for the domain Confidence 
that appears in the observation measure. Staff can develop their 
own descriptors based on the specific objectives of their project. 
Numerical values can be auached to the descriptors to facilitate 
data analysis (see Chapter V for strategies for analyzing data). 

Descriptors for Confidence 
Level 1 

Appears shy and unsure of own abilities; does not volunteer 
information; reluctant to answer if called cxi; seems overwhelmed 
by most new tasks; very pleasant person who needs a lot of 
support. 
Level 2 

Acts confidently when handling familia: information and seems 
willing to take risks; freely talks about himself/herself and shares 
information with the class; needs additional challenges to grow 
and learn. 

Levels 

Extremely confident; has a tendency to take over the class and 
sometimes shows off, causing slight resentment in the other 
students; requires challenging tasks and needs to be in groups 
where he/she cannot dominate easily. 

Similar descriptors can be developed for the other two non- 
linguistic domains in the measure, Participation and Transfer to 
real life. 
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Example O 



Observation Measure for 
Non-linguistic Domains 



Date 



Teacher 



Unit 



Class 



Confldonce 



Participation 



Transfer to real life 



Descriptors: 

• Appears shy and unsure of 
own abilities 

or 

• Acts somewhat confident 
when on familiar ground 
but is reluctant to take 
risks 

or 

• Sometimes acts overly 
confident, intimidating 
other learners 



Descriptors: 

• Participates and interacts 
with other members of the 
group 

or 

• Appears overly dependent 
on one or two friends 

or 

• Uses the group as a 
resource and acts as a 
resource 

or 

• Likes to work indepen- 
dently most of the time; 
does not enjoy group work 



Descriptors' 

• Does not yet use literacy 
skills outside the class 

or 

• Practices at home 

or 

• Integrates new informa- 
tion and skills into daily 
life 

or 

• Brings questions from 
home to the class 
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Adults who are non-IKer- 
ate in English as well as in their 
own language need a great deal 
of time and special support so 
they can comprehend printed 
materials and exptess their 
views in writing. 
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i-anguage experience 
stories can be used as starting 
points for assessing reading and 
writing for learners at lower lev- 
els of proficiency. 



Developing Literacy Through Telling and Writing Stories 

One of the most challenging tasks faced by Hteracy project staff 
is introducing non-readers to literacy. Aduks who are non-literate in 
English as well as in their own language need a great deal of time and 
special support so they can comprehend printed materials and express 
their views in writing. This section shows examples of activities 
designed to facilitate the learners' literacy development. Instruments 
and procedures are presented for holistic assessment of writing 
samples. 

Here's What to Do 

Many useful strategies are available for helping learners acquire 
literacy. Teaching the alphabet and moving to whole words through 
phonics is only one way — and not the best way according to most 
educators. A more effective approach would be to begin by helping 
learners read words they already know such as their own names, the 
names of their children, traffic signs, store marquees (K-Mart), and 
product logos (Coca-Cola). Activities based on environmental print 
can provide an excellent starting point for developing literacy. 

For teachers who believe that instruction in phonics is an impor- 
tant part of introducing learners to literacy, phonics activities can be 
integrated easily into authentic reading and writing activities. For 
example, to help her students move from oral language to literacy, 
Dean (1992) suggests using popular songs as a starting point for 
literacy activities. She uses the Mexican song "La Bamba'* to help 
learners associate sounds with print and to demonstrate that many 
syDables in Spanish consist of similar letter combinations (for ex- 
ample, la\ bam'ba\ ma-ri-ne-ro) . 

Another promising way to introduce reading to non-literate 
learners is the Language Experience Approach (LEA). LEA represents 
both a philosophy of teaching and learning and a method for connect- 
ing oral language to literacy. It involves a literate person, such as a 
staff member or another learner, transcribing a story that a learner or a 
group of learners has created based on an indivi^ or shared experi- 
ence. The stories are then read aloud and shared with group members. 
After the learners become comfortable with the content and language, 
the stories can provide the basis for group discussions, additional 
reading and writing activities, or in some cases, language practice. 
Because the language used in these stories comes from the learners, 
the vocabulary is familiar, the stories are reflective of the learners' 
experience, and the sentence structures are comprehensible. Some 
literacy projects evaluate the learners' progress in acquiring language 
by assessing the LEA stories they generate. 

Assessment Strategies 

Language experience stories can be used as starting points for 
assessing reading and writing for learners at lower levels of profi- 
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As learners new to lit- 
eracy make progress in the 
project, assessment may focus 
on showing how the range of 
literacy materials increases as 
learners become more efficient 
readers and writers. 



To assess writing ability, 
many projects use holistic writ- 
ing assessments that evaluate 
the learners* work from a variety 
of perspectives. 
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Because progress in writ- 
ing is often difficuK to observe 
from day to day, FELP staff may 
want to keep a collection of 
learners' work in individual port- 
folios. 



ciency. Otlier assessment strategies may include interviews that ask 
readers to talk about what they like to read and how they derive 
meaning from print. Also helpful are surveys in which learners are 
asked to list the kind of reading and writing they do in the course of 
their daily lives (for example, making grocery lists, checking televi- 
sion viewing schedules, reading the BiblCy observing traffic signs, and 
scanning school notices). 

As learners new to literacy make progress in the project, assess- 
ment may focus on showing how the range of literacy materials 
increases as learners become more efficient readers and writers. At the 
end of an instructional cycle, even beginners may be able to read signs 
and ads as well as simple notes, cards, stories, and poems. By observ- 
ing reading behavior and asking learners to describe the strategies they 
use for deriving meaning, staff can document increases in the range of 
literacy practices in which learners engage. 

Learners also will increase their range of writing abilities. Slowly 
they will write letters more easily and require less effort to get words 
down on paper. For example, a learner may move from writing her 
name in big letters on a name card to signing a birthday card to be sent 
to a friend. As learners become more comfortable expressing their 
ideas, their writing may show a strong voice that evokes the reader's 
emotion. A mother may write "My children far away. I very sad. I 
wish we together/' Although the structure of these sentences is not 
grammatically correct, the piece has a focus, shows logical coherence, 
and clearly communicates the loss the writer feels. To assess writing 
ability, many projects use holistic writing assessments tliat evaluate 
the learners' work from a variety of perspectives. 

For holistic scoring to be reliable, each piece of writing needs to 
be judged independently by at least two raters who provide reasons for 
their scores. If staff members have limited experience in assessing 
writing, they may need several opportunities to reflect on and discuss 
their judgements before they can reach a consensus. 

Example 10 is an assessment instrument designed to evaluate 
writing samples according to the dimensions of (1) authenticity, voice, 
and engagement of the reader, (2) focus, organization, and develop- 
ment; and (3) sentence mechanics and language. The example was 
adapted from an instrument that appears in Soifer and others (1990). 
Chapter V presents strategies for collecting, analyzing, and reporting 
data collected with Example 10. 

Because pn^gress in writing is often difficult to observe from day 
to day, FELP staff may want to keep a collection of learners* work in 
individual portfolios. Writing samples for the portfolio may be chosen 
and evaluated at key points during the instiuctional cycle. At specified 
intervals (for example, once every three weeks), staff members might 
identify several examples of the learners' writing that were produced 
during that time period. Staff could share their impressions as they 
review the writing samples. Staff members might invite learners to 
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Example 0 



Writing Assessment Instrument 



A. Authenticity A^oice/engagement of the reader 


4 = Expression strongly reflects the writer's emo- 
tional and/for intellectual involvement in the topic. 
Strongly engages the attention of the reader. 


Score/ 
Comments: 


3 = The writer is engaged in the topic and engages the 
reader. 


2 = Uninteresting; not engaging; perfunctory. 


1 = Writing seems to be a mechanical exercise, 
ingless expressions. 


B. Focus/organization/'development 
Comments: 


4 = Focuses on one main idea. Has clear beginning, 
middle, and end. Well-organized and well- 
ueveiopcu mrougn examples. 


3 = Focused and organized but may have a flaw in 
coherence or incomplete closure. Incomplete 
development. Explanation is strongly implicit. 


2 = Lack of clear focus, organization, or develop- 
ment. Narrative, but no explanation. 


1 = Disorganb.ed; underdeveloped; unconnected 
generalizations. 


C. Sentence mechanics/language 

Score: 
Comments: 


4 = Few mechanical, usage, or sentence errors. 
Language used with fluency and variety. 


3 = Some minor mechanical usage or sentence errors. 
Language used competently to express ideas. 


i 

2 = Enough usa^e errors to attract attention away 
from the content Sentences understandable, but 
unconventional. 


1 = Language and mechanical errors impair meaning. 



Source: Adapted from Soifer, Rena el al. The Complete Theory to Practice Handbook of Adult Literacy: Curricu- 
lum Design and Teaching Approaches. New YOTk; Teachers College Press, 1990, pp. 178-179. 
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Ul]^^ Because reading and writ- 
ing involve thinking processes 
that are not always observable, 
teachers need to be open to sur- 
prises and develop strategies for 
finding out how learners are 
thinking as they read and write. 
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In assessing the progress 
of beginning literacy students, it 
is important for staff to be aware 
of the cyclk^al and recursive na- 
ture of literacy development. 



select what they regard as their best or favorite piece of writing and 
talk about their progress in one-on-one conferences with staff. 

Assessing the Progress of New Readers and Writers. Assessing 
the progress of beginning readers and writers can be a tricky undertak- 
ing. Three things need to be kept in mind: First, because reading and 
writing involve thinking processes that are not always observable, 
teachers need to be open to surprises and develop strategies for Finding 
out how learners are thinking as they read and write. Second, reading 
and writing development are not linear processes; rather, literacy 
develops in fits and spurts. As a result, a learner might apf)ear to be 
fairly high on a reading and writing scale one week and lower the 
next. Third, learning how to read and write are complex processes thai 
require special skills and strategies — for example, being able to 
connect meaning to print and understanding that writing represents 
sp^eech. 

Literacy also involves sub-skills such as understanding the 
relationship between letters and sounds (phonics) and the ability to 
form the letters of the alphabet. Another important literacy skill is 
sometimes referred to as making meaning — that is, the ability to make 
sense of written words or to express thoughts in writing. The checklist 
shown in Example 1 1 is designed to assess the meaning-making 
strategies associated with literacy. 

In assessing the progress of beginning literacy students, it is 
important for staff to be aware of the cyclical and recursive nature of 
literacy development. Staff members should not be surprised if a new 
reader understands quite a bit of one story but becomes confused or 
stymied by another. Similarly, a writer may feel inspired by a particu- 
lar topic, yet have nothing to say on a different theme. 

Understanding some of the factors that help or hinder compre- 
hension and writing proficiency helps staff to assess literacy progress. 
The learners* background knowledge, goals, and interests influence 
how well they are able to deal with a particular text. Thus, staff need 
to observe and listen to the learners to get a sense of the kinds of 
topics they may want to read and write about. Similariy, staff should 
provide a wide range of literacy activities (for example, listening to 
songs, labeling photographs, illustrating a story, or reading about 
others in the class) so that they can respond effectively to the students' 
different learning styles. Staff also should look at many different 
examples of what learners are able to read and write so they can get a 
more complete picture of the students' literacy development. 

The checklist (Example 11) provides descnptions of some of the 
strengths that learners exhibit as they go about making meaning in 
reading and writing. It illustrates some of the learners' characteristics 
as they move from being relatively unfamiliar with literacy to becom- 
ing independent readers and writers. Not all learners will exhibit all of 
the descriptors in the instrument. As staff members read and write 
with their learners, they should check the descriptors that characterize 
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Checklist for Assessing Beginning 
Literacy and Emerging Literacy 



Learner's name: Dale: 

Part I: Assessing Reading 

A, As literacy starts to emerge, the new reader: 

1 . □ Is aware that prim carries meaning but only in a very vagiae sense. 

2. □ Can tell prim from non-prim (e.g., decoration) but insists that she cannot read anything. 

3. □ Can tell the shape and form of English writing from other types of writing but resists 

making a guess about the meaning of a word. 

4. □ Recognizes her own name and a few sight words but resists predicting what a new word 

might mean. 

B. As literacy continues to develop, the new reader: 

1 . □ Can interpret print that is part of the environment, such as reading a stop sign, recogniz- 

ing a Coca-Cola can, pointing out tlie McDonald's logo, or interpreting a No Smoking 
sign, as long as a great deal of support is given. 

2. □ Is able to read familiar signs, labels, and logos as long as they appear in their natural 

form (e.g., rice printed on bag of rice); is willing to experiment and venture some 
guesses. 

3. □ Recognizes familiar words when v/ritten on paper, as long as some context is provided 

(e.g., a picture of a bag of rice with rice written next to it). 

4. □ Can read short familiar texts, such as simple language experience stories thai arc read in 

a group. 

5. □ Employs reading strategies that allow her to move through familiar text independently. 

Part II: Assessing Writing 
A, As literacy emerges, the new writer: 

1 . □ Is not yet aware of the functions and uses of writing. 

2. □ Is aware that writing carries meaning, but is not yet ready to try to write. 

3. □ Copies letters and words but is not able to read what she writes. 

4. □ Tries to write something, even if only a few scribbles; talks about what she has written; 

understands that writing is used to express ideas. 

5. □ Writes her own name independently, as well as some other key words; however, writing 

still seems like a mechanical exercise. 
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Checklist for Assessing Beginning Literacy and Emerging Literacy (continued) 



As literacy continues to develop, the new writer: 

1 . □ WriiesS or copies words and is intftrested in what the words mean; tries out some of 

her own ideas as long as some support is given; may copy a word and use it to label 
a picture or photograph. 

2. □ Writes a few words or sentences independently without copying; writing may be 

uneven and meaning not always clear, 

3. □ Experiments with writing on her own; may write a few words about herself or may 

write about what she sees in a photograph. 

4. □ Experiments further with expressing ideas; writes a few words in dialogue journals 

or attempts to write parts of a language experience story; may need a great deal of 
support. 

5. □ Continues to write notes in dialogue journals and writes some parts of language 

experience stories independently; writing shows beginning elements of organiza- 
tion. 

6. □ Connects ideas and thoughts in writing; may write short description of a picture or 

an event; writes short description of self or of family; spelling and sentence struc- 
ture still very uneven. 
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a particular learner. They can then compare the checklists of several 
learners to determine how the learners as a group are developing from 
emerging to independent literacy. Staff members also can compare 
checklists on several learners and summarize the data at the end of an 
instructional cycle. 

Summary 

Adult learners have their own reasons for coming to class, only 
some of which have to do with acquiring English and developing 
literacy skills. Effective assessment approaches need to take into 
account the learners' needs and goals and develop ways of showing 
them that they are making progress. 

Alternative approaches to assessment enable staff to create 
project-based instruments and involve learners in developing the 
curriculum and designing tlie evaluation. The instruments may include 
observation measures, protocols for interviews and focus groups, 
learner profiles, and perfomiance samples. Staff will want to use and 
field test these instruments so that they can measure progress reliably 
over time. ITie data gathered with the instruments can then be aggre- 
gated into summary formats and, where appropriate, quantified so that 
the information can be used to answer a variety of questions about the 
project. 

FELP staff who use alternative assessments that arc project- 
based and learner-focused have access to more valid and useful 
information than that which can be obtained from most standardized 
tests. To improve the use of alternative approaches to assessment and 
evaluation, project staff face the challenge of defining their vision of 
success and developing instrument? ♦hat capture the changes that take 
place as language-minority adults and their children become involved 
in family literacy. 
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SURVEY OF USAGE 

Chapter IV. Assessing Ongoing Progress: Are We Progressing? 

This resource guide must be helpful to you, the FELP director or evaluator, in order to be considered 
a success. In accepting the field test edition of the guide, we hope that you agree to participate in this 
usage survey. We plan to revise the guide based on your comments and criticisms; it then will be 
published for a broader audience. Please share with us your critical comments. After reviewing the 
chapter, use the scale below to indicate whether the guide is writtan clearly; the scale indicates that 
it is 

1 . Clearly written and understandable, the format is clean; 

2. Generally clear, but some wordings or format could be improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at all clear, not at ail understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. In each of the same areas, please use the scale to indicate that the guide has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal; or 

4. Information that I don't need and won't use. 



Rate clarity Area to be rated 

1 2 3 4 Format of the chapter 

1 2 3 4 Wording of the chapter 

Not applicable The general ideas and information presented 

12 3 4 Characteristics of alternative assessment 

1 2 3 4 Getting acquainted: Learner profile/observation card 



Rate information 
Not applicable 
Not applicable 
12 3 4 

12 3 4 

12 3 4 



Family life: 



Questionnaire to identify home literacy 12 3 4 

Self-assessment of literacy activities 12 3 4 

Questionnaire recording home literacy 12 3 4 

Self-assessment for setting goals 12 3 4 

Questionnaire to assess behavior changes 12 3 4 



12 3 4 

12 3 4 

12 3 4 

12 3 4 



Learning to read: Observation measure 
Learning to write: Observation measure 
Self-esteem and literacy: Performance chart 
Telling and writing stories: Writing assessment 



12 3 4 

12 3 4 

12 3 4 

12 3 4 



Thank you for your help. If there are other areas that you feel should be included in this chapter, or 
other comments about the chapter that you would like to make, please feel free to do so. You do not 
need to provide your name. 
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^ Please return your completed survey. You may fold, staple, and stamp this page - the address is on the back 
side. If you prefer, you may FAX it to Daniel D. Holt, Bilingual Education Office at (916} 657-2928. We also 
would appreciate receiving a copy of any alternative assessments you create as a result of this guide. 



Fold here 




Daniel D. Holt 

Bilingual Education Office, California Department of Education 

PO Box 944272 
Sacramento, CA 94244-2720 



Fold here 
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CHAPTER V 

COLLECTING, ANALYZING, 
AND REPORTING 
ALTERNATIVE 
ASSESSMENT 
RESULTS 




Sal Gelardi 
Educational System Planning, Inc. 
Woodland, California 

This chapter deals with collecting, analyzing, using, and repwrting data 
obtained with four alternative approaches to assessment and evaluation: 
surveys*, performance samples, intei"views, and observation measures. A 
variety of different analyses, techniques, and procedures are presented 
which can be used with each of the alternative approaches. However, the 
strategies presented for the four instruments also may be utilized with other 
types of alternative assessments. The chapter illustrates how alternative 
assessment data can analyzed, interpreted, and reported according to the 
evaluation requirements of ESEA, Title VII and the special characteristics 
of the Family English Literacy Program (PELP). Although the chapter 
focuses on FELP projects, the information could be adapted for other 
literacy projects for language-minority adults. 

The chapter is divided into four sections, each of which addresses a differ- 
ent alternative approach. Each section deals with the four key aspects of 



* See the Glossary on pages 114-115 of this publication for defini lions of importanl terms 
that appear in this chapter. 
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FELP staff who are 
interested in more information 
about alternative approaches 
to assessment and evaluation 
are encouraged to consuH with 
staff of the ESEA, Title Vll-funded 
Evaluation Assistance Center 
in their region. 



^ 

One of the important 
advantages of alternative assess- 
mer ^proaches is that they 
offer staff the opportunity to 
involve learners m the assess- 
ment process. 



managing the data obtained from the alternative approaches: (1) 
collecting the data; (2) analyzing tlie data; (3) using die data; and (4) 
reporting the data and findings. Strategies for data collection are 
provided for each approach. 

Techniques for analyzing, interpreting, and reporting alternative 
asr^essmcnt data are presented in the chapter with increasing levels of 
sophistication. The first section of the chapter deals with basic tech- 
niques for dealing witli the results of surveys. More complex strategics 
arc illustrated in the last section on observation measures. In general, 
however, techniques presented in one section may be used with the 
other alternative assessment instnjments presented in the chapter, 

Three audience^ are considered in the use and reporting of 
alternative assessment data: FEl.P project directors. FELP staff 
members, and the staff in the Office of Bilingual Education and 
Minority Languages Affairs (OBEMLA). U,S, Dcpaitmcnt of Educa- 
tion. Project directors may use the data for planning and improving 
program activities. FELP staff members may be interested in diagnos- 
tic and monitoring uses; OBEM^I^A staff miglu be interested in how 
the data can be summarized in the annual evaluation report and final 
pcrfonnance report required by ESEA, Title VIL 

FELP staff who want more infonmation about alternative 
approaches to assessment and evaluation are encouraged to consult 
with staff of the ESEA, Title Vll-funded Evaluation Assistance Center 
in their region, They may also wish to review other resources such as 
Qualitative Evaluation Methods by Patton (1980). 

Consistent witli tlie Planning Process Model presented in Figure 
1 , Chapter IIv data obtained through alternative assessment approaches 
are useful in the pre-planning, planning, and evaluation phases of 
FELP projects. Alternative assessment insU'uments arc particularly 
useful in giving staff information necessary for (1) planning a program 
based on the needs of adult learners; (2) aligning curriculum according 
to the results of learner assessments; (3) monitoring learners* progress 
towards attaining project objectives; and (4) involving learners in 
determining their progress toward mutually agreed upon goals. (See 
chapters I and II for more on objectives.) 

One of the important advantages of alternative assessment 
approaches is that they offer staff the opportunity to involve learners 
in the assessment process. Learners may be involved in such tasks as 
identifying topics that will be assessed, monitoring their progress 
toward attaining project goals, or negotiating their rating on a perfor- 
mance sample. As parents collaborate in tlie assessment process, they 
take more responsibility for their own learning and the project in 
general. 

Surveys 

Surveys may be used to collect information for the pre-planning, 
planning, and evaluation phases of the plarming model presented in 
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Chapter Ih In tlie pie-plarming phase, survey data are used to identify 
the needs of potential participants and to desigxi objectives for the 
project. During the planning ptiase, survey dati>, may be used to 
develop curriculum and instructional activities. This section addresses 
the use of survey results in the evaluation phase of the project. The 
information presented is based on the Needs Assessment Survey 
presented in Chap'^r III, 



Example Q 



NEEDS ASSESSMENT SURVEY 



N-imc _ [ Telephone f > 



NMiivc Lanftuagc 

1 Mow many adulis live in >our apiruncm ' 

2 How muiy children in your apanmcm aiiend Lincoln UniHcd School Dtnnci? 
I . 'cnufy individual schools by name) 

3 How many adulu turrcnUy aticnd GAIN program English classes'^ 
■I Where do iticy aiicnd cliiscs* ic g . Delia College, adull school) 

5 If there an; adulis livinp in ihis apinincni who an; nor aiicnding GAIN English 
classes, would ihcy aiicnd (lAiN linglish cLisscs if ihcy were held in your 
apanmcni complex^ 

fi How many men would jio' 

" How many women would go ' 

K Would ihcse adulis aprvc lo auci>d tUs;. 4 ho[irs a day. 5 days a N^cek^ 

9 Would Ukv nrcd lo ha\c someone outside of ihc lainrly lake care of UKir 
childTtn dunng cliss"^ 

10 How many children need child care * 

1 1 Should class be held in the morning or in ihc aficmcon ' 

12 Is ii a good idea lor us lo try to pet our cljLS.ses approved for GAIN, or sixwid 
we have our own liierjts pm^rani ' 

n If we can.->ot get our KnglKh cUvscs approved for GAIN, would you aucfxl 
inyway ' 

14 How many hnxin per d.i> should classes he held'' 

15 How many days a week should classes be held'' 

16 Wha( would you like to loam* 

a. English d How the icbools opcr^ies 

b Mow to get a job c How to become a ciozen 

c. How to be a gwxl parent 

17 What IS your level of English language skills" 

a. Oral skills. « I » Low . : » Medium ( 3) High 
b Redding and wntm^ skills 1 1 ) Low i2)Mcdiui'n (3)High 
18. How long have you been in the I niied Stales? 

19 How many children in this I Jm»l\ are panictpating in ESEA. Title VM. programs? 



Ftmlly Engltih Ltt*f«cy ProjMrl • Uftcein UnifM S«fw*t tXttrkI • Moditcfi.CAlHomla 



Coltoctinie Survey Data for Evaluation 

The responses to each question on the survey should be tabulated 
to determine the range of responses and the number of respondents 
who selected each response. For example, the first question on the 
survey asks the respondent to indicate the number of adults who live in 




9 
K) 
II 



n 

14 
15 
16 



I7a 
17b. 

18. 

19 



Staff may find it helpful to 
organize the data analysis by 
posing questions that focus on 
key aspects of the project. 



his/her apartment. The range of responses for this question would 
consist of the smallest and largest number of adults living in one 
apartment. A frequency count would indicate how many respondents 
selected each response. Data should be tabulated similarly for each 
item in the survey. The range of responses should be tailored for each 
question according to the type of response choices (for example, yes/ 
no, names of schools, or morning/afternoon). After the data have been 
compiled, the total frequency count should equal the number of people 
who responded to the survey. 

Analyzing Survey Data 

The results of the needs assessment can be analyzed to answer 
questions related to planning or evaluating the project. The process of 
analysis consists of reviewing the compiled data and making judg- 
ments about the needs of the learners or about how the project can be 
improved. Staff may find it helpful to organize the data analysis by 
posing questions that focus on key aspects of the project. Example 1 
contains sample questions that staff members might use for organiz- 
ing, analyzing, and using data that are collected with a survey. The 
corresponding questions from the Needs Assessment Survey are 
identified in parentheses. 



Exiimple 1 



Sample Questions for Data Analysis 

1 . How many aduHs living in this community (male and female) 
are eligible to participate in the project but are not enrolled 
currently in English-as-a-second-language (ESL) classes? 
(questions 1,2,6. and 7) 

2. How many adults living in this community are interested in 
attending ESL classes? (question 5) 

3. What are current and potential participants' needs for child 
care? (questior^s 9 and 10) 

4. What are current and potential participants' preferences 
regarding the scheduling or classes? (questions 8, 1 1 , 14, 
and 15) 

5. What are cun'ent and potential participants' preferences 
regarding the content of instruction? (question 16) 

6. What are current and potential participants' 
speaking.reading.and writing skills? (questtons 17a and 17b) 



Staff Use of Survey Data 

Survey data may be used by staff to determine how well the 
project is meeting the learners' needs. Given the high turnover of 
participants in many FELP projects, a survey such as the Needs Assess- 
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Given the high turnover of 



participants in many FELP 
projects, a survey such as the 
Needs Assessment Survey should 
be conducted on a regular basis 
to ensure that project services 
are based on current learners' 
needs. 



During regular project 
meetings, staff can consider the 
results of surveys to determine 
the extent to which they are 
addressing the learners' needs 
and to deaJ with other issues that 
might have been identified when 
the survey was conducted. 





ment Survey should be conducted on a regular basis to ensure that 
project services are based on current learners' needs. For example, the 
findings from questions 1-7 can be used to identify important back- 
ground information about the participants. This information also can be 
used to help staff determine the effectiveness of their recruitment 
efforts in various neighborhoods. In addition, it can help staff evaluate 
the suitability of current class locations and identify more convenient 
sites such as community centers or apartment complexes. 

During regular project meetings, staff can consider the results of 
surveys to determine the extent to which they are addressing the 
learners' needs and to deal with other issues that might have been 
identified when the survey was conducted. Such discussions encble 
staff to improve current services and plan future activities. Survey data, 
collected annually or more frequentiy, can be used to answer questions 
related to developing annual evaluation reports (for example, **What 
needs have been met?" **What new needs have been identified?" What 
progress has been made in meeting project objectives?"). 

Reporting Survey Results According to Questions 

Methods of reporting the results of surveys depend on how the 
data will be used. If the results are to be used for project evaluation, 
they could be organized around the evaluation questions shown in 
Example 1 in the preceding section. The data corresponding to each 
survey question may be (1) described in brief sentences (for example, 
"Over half (55 percent) of the respondents indicated that they need to 
have someone outside the family take care of their children while they 
attend class" or "Approximately 30 children will need child care") or 
(2) illustrated in tables depicting the range of responses and number or 
percentage of responses. 

Example 2 (Table A) illustrates how data from Question 1 of the 
Needs Assessment Survey may be presented in a table as part of an 
annual evaluation report or final perfonnance report. The data may 



Example 2 



TABLE A 

Number of Adults Living in One ApartriDent 



Number of 
Adults 


Numbsr/Percentage 

Responses 
Total Responses='25 


3 


8 (32%) 


4 


11 (44%) 


5 


6 (24%) 
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show evidence that the project is successful or is in need of modifica- 
tions. Table A verifies that a majority of adult participants live close to 
each other in apartments, which suggests that holding classes in or 
near the apartment complex would be convenient for the participants. 

The reporting of survey results should focus on how staff can 
improve the project. The conclusions or recommendations should 
provide answers to questions that staff and learners have posed about 
the project. Example 3 contains a table and narrative that summarize 
the responses to Question 3 in Example 1 ("What are the current and 
potential participants' requirements for child care?'*) and present 
findings and recommendations. 

Performance Samples 

Performarce samples are illustrations of learners' work in 
selected tasks. As one kind of performance sample, a writing sample 
can provide valid, authentic representations of learners' progress in 
attaining project objectives. Because they can be linked directly to 
project activities, writing samples become natural extensions of the 
instructional process. Leamers can produce writing samples in English 
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TABLE B 

Summary of Respondents' Child Care Needs 



Item 


Currently Serving 


Need to be 
Served 


Conclusions 


Number of adults 
who need child care 
to attend class 


25 


35 


Ten additional adults require child 
care services. 


Number of children of 
adult participants 


30 


45 


The current chikl care program at 
Delta College is designed to serve 
25 children. It is already over- 
crowded. 



The original survey identified 25 adults who needed child care services in order 
to attend classes. Although the Delta College ChiW Care Facility couW accommodate 
only 25 chikJren, 30 chiWren received sen/rces during the first week of the project with 
the help of volunteers recmited by the project staff. After four weeks, ten new learn- 
ers requested child care services in order to attend class. Because the Delta College 
facility is unable to handle additional chikJren, staff recommends that another class- 
room be found for the additional children. 



The reporting of survey 
results should focus on how staff 
can improve the project. 



or their native language. As learners develop examples of their written 
work throughout the year, they and the staff can identify writing 
samples that demonstrate strengths as well as areas that need improve- 
ment. 

This section presents strategies for collecting, analyzing, using, 
and reporting the results of alternative approaclies to assessing writing 
samples. Strategies refer to the Writing Assessment Instrument shown 
in Chapter IV. 



gjcampie ^ ^'i 
Writing Assessment Instrument 



A. Authcnticity/voicc/cngagcracnc of the reader 


4 a Expreision strongly refleco the writer*! crao- 
dorul and or inteUectual involvement in the copic. 
Stronily enfates the toentkm of the retder. 


Conumeats: 


3 - The wriier is eofifed ic the topic Mid eap^ the 
reader. 




2 » UoioieresiQf ; tKX eogaginf; perfuoctory. 




1 ■ Writini aeemj to be a mechanica] exercije. 
Markad by cUcfaei. hazy generalizadoni. meao- 
trtfteu expresflons. 


3. Foctu/ofgimzadon/devtlopment 


I » Focuses on ooe main idea. Has clear beginning, 
middle, and end. Well-orianiad and well- 
developed through exanplet. 


Comments: 


3 * Fociued and organutd but may have a flaw m 
coherenoe or incompleR closure. Incoropleie 
development Expl^judon is strong ty unpUat 




2 « Ljck of clear focu^ organizadon. or develop- 
ment. Namdve, but no explanaboo. 




1 Disofgamzed; underdeveloped; unconnected 
genenlizacofts. 


C. Sentence roechanics/Iaiiguage 


4 a Few mechanical, uuge, or sentence enon. 
Language used with fluency and variety. 


Comments: 


3 » Some minor mechanical usage or sentence emxi. 
Language used compesendy to expftss ideas. 




2 » Enough usage emon to anna attention away 
from the content. Seatenoes undentandable, but 

uQcoaventiotutl. 




1 » Langua^ and mechanical eiron impair meaning. 



Source: Adapted from Soifer, Rena et ai. The Complete Theory to fracace Handbook o/AJuU Literacy: Curricu- 
lum Dtsxgn and Teaching Approaches. Ne'" Yotfc Tcichen CoUege Prwi, 1990. pp. 178-179. 



The Writing Assessment Instrument uses a rating scale to evalu- 
ate the learner's perfonmance. A rating scale is a technique for assess- 
ing an area of interest (for example, reading ability or self-confidence) 
on a scale of intervals usually designated by numbers or descriptors or 
both. Rating scales are frequently employed with alternative assess- 
ment instruments to differentiate ability or perfonnancc levels. Ratings 
are based on the learner's performance relative to narrative explana- 
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By developing project- 
based descriptors and associated 
rating scales, staff can improve 
the validity and reliability of the 
Instrument and enhance the ac- 
curacy and consistency of as- 
sessment and data collection. 



As a natural extension 
of a learning activity, a writing 
sample serves both instructional 
and assessment purposes, 
thereby eliminating the time 
and energy required for separate 
tests. 



lions (descriptors) of each individual rating on the scale. Numerical 
values can be attached to the descriptors to facilitate data analysis. 
Sometimes ratings are combined with narrative comments and placed 
in a performance chart to provide for a more comprehensive analysis 
(sec Example 9 in Chapter IV). 

The Writing Assessment Instrument allows staff to judge a 
writing sample using four-point (1-4) rating scales. A descriptor is 
provided for each of the four ratings. A set of ratings was designed for 
each of three aspects of writing: authenticity, focus, and mechanics. 
For example, a rating of 1 for authenticity has a different meaning 
from a rating of 1 for focus or sentence mechanics. 

FELP staff could adapt the instrument by developing descriptors 
that would be appropriate for the learners and curriculum in their 
project. By developing project-based descriptors and associated rating 
scales, staff can improve the validity and reliability of the instrument 
and enhance the accuracy and consistency of assessment and data 
collection. The descriptors should be arranged in logical sequence 
with clear definitions for each rating. 

As a natural extension of a learning activity, a writing sample 
serves both instructional and assessment purposes, thereby eliminating 
the time and energy required for separate tests. By collecting and 
assessing writing samples according to predetermined intervals and 
with agreed-upon rating scales, staff members can make judgements 
about the learners* growth as well as progress that they are making 
towards attaining the objectives of the project. To use rating scales 
effectively, staff members need to have a thorough understanding of 
the descriptors. Staff members can achieve this understanding by 
receiving training in the rating procedure and by collaborating to 
design the descriptors for the ratings. 

Collecting Data from Writing Samples 

Example 4, Table C, shows how data on learners' writing 
samples can be collected systematically. Data obtained through this 
procedure can be used to monitor learners' progress in attaining the 
objectives of the project. 

The Qass List helps staff to record assessment results of writing 
samples according to predetermined intervals such as weekly, 
monthly, or quarteriy. Staff can use the rating scale to judge learners* 
writing or ask learners* to judge their own work. To accommodate the 
learners* language prof iency levels, staff can simplify the descriptors 
so that learners can read a'^.d use the rating scale for their self-assess- 
ments. In this way, assessment becomes an instructional tool that 
enables learners to recognize and judge effective writing. Portfolios 
can be used for keeping a record of selected examples of learners* 
writing. Although learners may write on a daily basis, staff can have 
learners select on a specified schedule their best work for inclusion in 
the portfolios. 
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Portfolios can be used for 
keeping a record of selected 
examples of learners' writing. 
Although learners may write on a 
daily basts, staff can have learn- 
ers select on a specified sched- 
ule their best work for inclusion 
in the portfoltos. 



Analyzing Data with Frequency Counts 

Simple frequency counts provide a useful way to analyze data 
collected witli rating scales. Formore sophisticated data analysis, staff 
should seek the assistance of a professional evaluaton The analysis of 
ratings of writing samples can be facilitated by using a class list like 
the one shown in Example 4. Used with the Writing Assessment 
Instmment, the list helps staff document the ratings according to the 
four descriptors (1-4) in each of the three aspects of writing: authentic- 
ity, focus, and mechanics. The list provides space for ratings taken at 
three different times, First Testing, Second Testing, and Third Testing. 
This fontiat enables staff to moriitor the progress of each individual as 
well as the total class. 

The Qass List contains the ratings of a partial number (10) of 
participaiits in a FELP class. The data represent the three assessments 
conducted in October, January, and April. The numbers under A, B, 
and C represent the assigned ratings according to the descriptors 
specified in the instmment. The list can be used internally by the staff 
or provided as a table in an annual evaluation report or final perfor- 
mance report. 



ilxainple 4 



TABLE C 

Class List: Results of Assessments of Writing Samples 

Instructions: Record date of ♦he assessment and enter the rc ing (1-4) under each letter: 

A « Authenticity A^oice /Engagement of the Reader; B « Focus/Organization/Devebpment; 
C » Sentence Mechanics/Language 







First TQStir)g 




Second T0Sting 




—-1 

Third Testing 


Name or I.D. No. 


Date 


A 


B 


C 


Date 


A 


e 


C 


Date 


A 


e 


C 


B. Lopez 


10/25 


1 


1 


2 


1/15 


1 


2 


2 


4/2 


1 


2 


3 


J. Garcia 


10/15 


1 


1 


1 


1/15 


1 


1 


1 


4/2 


1 


1 


1 


M. Matelio 


10/15 


1 


1 


1 


1/15 


1 


2 


2 


4/2 


1 


2 


3 


K. Turner 


10/15 


1 


2 


3 


1/15 


2 


3 


4 


4/2 


2 


1 


2 


R. Diaz 


10/15 


2 


1 


2 


1/15 


2 


1 


2 


4/2 


2 


1 


2 


B. Ruiz 


10/15 


1 


2 


1 


1/15 


2 


2 


2 


4/2 


2 


2 


3 


A. Rodriguez 


10/15 


1 


1 


1 


1/15 


2 


1 


2 


4/2 


1 


3 


3 


J. Fernandez 


10/15 


2 


1 


1 


1/15 


2 


2 


2 


4/2 


3 


4 


3 


C. Lopez 


10/15 


1 


2 


1 


1/15 


2 


4 


2 


4/2 


2 


4 


3 


B. Nunez 


10/15 


3 


4 


3 


i/ir 


3 


4 


4 


4/2 


4 


4 


4 
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The data in the Class List may be analyzed further by counting 
how many learners receive each rating for each aspect at each data 
collection point The tallies are converted to percentages because the 
number of learners generally varies from one testing interval to the 
next. Staff can interpret the impact of the project by examining the 
degree of change in percentages across the three testing intervals. This 
change can be illustrated graphically and documented in a matrix like 
the one shown in Example 5 (Table D). 

Table D provides a summary of the results of writing assess- 
ments conducted at thiee testing intervals (October, January, and 
April). It displays ratings for three aspects writing: authenticity, focus, 
and sentence mechanics. If this summary were included in an evalua- 
tion report, it should be accompanied by a narrative description of the 
progress and trends toward accomplishing project goals such as: 

The table reflects an increase in the percentage of learners who score at 
higher levels with each assessment, suggesting progress in attaining 
project objectives. The greatest growth took place in "authenticity." 

Using the Results of Assessfnents of Writing Samples 

Staff can use the results of writing assessments for planning 
instruction, monitoring progress, determining the degree to which 
objectives have been achiev^^i, and refining the curriculum and 
instruction. The results can help staff members form groups of learn- 
ers with similar needs and design appropriate instructional activities. 
Data can be analyzed to indicate the progress of individual learners as 
well as trends of the whole class. Assessment results also can show the 
relative effectiveness of various components of the project. 



Examples ' ' : - .. . . 
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TABLE D I 
Summary of Three Writing Assessments | 



Testing 


Sentenc0 M0€hanics 


Focus 


Authenticity 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


First«Oct. (n-125) 


80% 


18% 


2% 


0 


687o 


20% 


12% 


0 


667o 


22% 


12% 


0 


Second»«Jan. (n»118) 


43% 


24% 


22% 


11% 


14% 


44% 


25% 


M% 


2% 


38% 


34% 


267o 


Third«April(n-105) 


20% 


53% 


237o 


47o 


0 


177o 


49% 


34% 


0 


177o 


29% 


54% 
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Staff may choose to iiv 
volve the learner in developing 
descriptors, selecting writings to 
be assessed, and ultimately 
evaluating their own writing. 



1^ 

When parents experience 
success at wrKing creatively 
about meaningful "topics, they can 
transfer their joy of writing to 
their children. 



Staff may review several writing samples by one learner to 
identify changes and trends in the learner's writing strengths. Such 
analysis can be accomplished with or without formal ratings. Learners' 
writing samples may also be shared among staff to gain insights into 
the overall effectiveness of instructional activities. When learners are 
promoted to higher levels, their writing samples can be passed on to 
their instructors to ensure that instruction is appropriate and uninter- 
rupted. 

Writing assessments are ini **viional tools as well as vehicles for 
appraising the learners' progress. When shared with the learner, the 
results provide immediate feedback that can lead to improved writing 
performance. Staff may choose to involve the learner in developing 
descriptors, selecting writings to be assessed, and ultimately evaluat- 
ing their own writing. Such self-assessments can help learners acquire 
an awareness of the characteristics of effective writing. 

When parents appreciate the importance of writing and other 
literacy behaviors, they can become more confident in supporting their 
children's language development When parents experience success at 
writing creatively about mcarJngful topics, they can transfer their joy 
of writing to their children. They may share their writing with their 
children, encourage their children to write about their own experi- 
ences, and even write stories together. 

Summarizing Writing Sample Assessments 
for Evaluation Reports 

Methods of reporting the results of writing assessments vary 
according to the purpose of the reports. One approach may be to 
present the Class List (Example 4) with an accompanying narrative in 
an evaluation report. The report could be part of an application for a 
continuation grant or used by project staff to plan additional classes 
and services. The data summarized in Table D (Example 5) could be 
included in a final performance report. To provide an interpretation of 
the data, staff can develop a narrative that summarizes changes that 
resulted from the project and progress made by the learners in achiev- 
ing the objectives of the project. 

Example 6 illustrates how staff m.ight report the results of three 
assessments of writing samples. The example, containing a table and 
narrative explanation, is based on the Writing Assessment Instrument. 
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Example 6 



Because Interviews can 
be conducted in the learners' 
native language, participants 
may be more willing and aMe to 
provide information to staff. 



TABLE E 

Learner's Growth In Writing (Authenticity) 



Testing 


AuthantlcHy 


1 


2 


3 


4 


First Testing (Oct.) 


66% 


22% 


12% 


0 


Second Testing (Jan.) 


2% 


38% 


34% 


26% 


Third Testing (April) 


0 


17% 


29% 


54% 



Sample Narrative 

Table E identifies the percentage of FELP participants 
scoring at each level (1 , 2, 3, 4) for each of the three assess- 
ment intervals (October, January, April). The percentage of 
learners who made progress from low levels (1 and 2) to high 
levels (3 and 4) increased substantially over the three testing 
intervals. The initial assessment in October identified 66 percent 
of learners at Level 1 and no learners at Level 4. By the second 
testing in January, however, only 2 percent of the learners 
scored at Level 1 and 26 percent scored at Level 4. By the end 
of classes in April, the third-interval testing indicated that no 
learners scored at Level 1, 17 percent scored at Level 2, 29 
percent at Level 3 and 54 percent at Level 4. Learners partici- 
pated in an average of three hours per week of writing instoic- 
tion and associated activities for a total of 72 hours during the 
24 weeks of classes. The assessment results demonstrate that 
learners have made progress in writing content that is authentic 
and engaging. 



Interviews 

Interviews can be used to collect detailed information for assess- 
ing and evaluating the project. An interview procedure may be utilized 
with individuals or adapted for small groups of people such as focus 
groups. This section will deal with the collection, analysis, staff use, 
and reporting of data obtained from focus groups. 

Interviews are particularly appropriate for FELP pmjects because 
participants may not have the literacy skills necessary for paper-and- 
pencil assessments. Furthermore, because interviews can be conducted 
in the learners* native language, participants may be more willing and 
able to provide information to staff. 
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An Important element of 
the focus group is the nature of 
the communication (verbal and 
noivverbal) between the facilita- 
tor and the respondents as ques- 
tions, answers, and comments 
are stated and clarified. 



Inicrvicws may be used independently or together with other 
measures to generate data for evaluating tlie effectiveness of instruc- 
tional programs and identifying successful components of the project. 
They can be conducted by an instructor, project coordinator, or 
professional evaluator. This section of the chapter is based on inter- 
views conducted with small groups according to a protocol which 
focuses on specific topics; hence, the title focus group. Protocol is 
defined as a specified set of p»"ocedures and questions designed to 
achieve the purposes of the assessment activity. 

Collecting IntcrvlGiv Dsts Through Focus Grou(^ 

Focus group data can be collected with a question-and-answer 
procedure in which a facilitator addressefi questions to group members. 
The responses are documented in writing or recorded on audio or 
video tape. Although the proceedings may be recorded by the facilita- 
tor, staff members may want to assign this responsibility to a separate 
individual to avoid burdening the facilitator. An important element of 
the focus group is the nature of the communication (verbal and non~ 
verbal) between the facilitator and the respondents as questions, 
answers, and comments are stated and clarified. Therefore, the facilita- 
tor should be sensitive to the cross-cultural and other interpersonal 
dimensions that characterize group activities. 

The protocol for a focus group should be established in advance 
and linked carefully to the goals and objectives of the activity. How- 
ever, the facilitator should not discourage responses that may deviate 
from the main purpose of the session. Such flexibility creates opportu- 
nities for the facilitator to assess a variety of related learner needs and 
outcomes such as language, social interaction, confidence, ?nd critical 
thinking, which are vcrj' difficult to assess through paper-and-pencil 
measures. These issues may be^ recorded in anecdotal form and ana- 
lyzed later when the tapes are reviewed. 

Interview data may be collected at intervals throughout the year, 
on a pre/post basis, or at the end of instructional units. The data can be 
analyzed and interpreted in terms of longitudiniU progress, pre/post 
changes, and project implementation and accomplishments. Generally, 
focus group data are collected and reported on the entire group, 
although data gathered on individuals during the session also may be 
useful. 

Analyzing Focus Group Data 

Analyzing focus group data can be a challenging task. Staff 
should strive for a balance between conr'ucting elaborate analyses of 
trivial points and ensuring that adequate attention is given to signifi- 
cant information that is generated. The importance attached to specific 
findings is determined by the evaluation questions, project objectives, 
and the intended audiences (for example, the school board, project 
staff, and OBEMLA). UlUmately, decisions regarding the complexity 
of the analyses should be guided by how the results are going to be 
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The goal of analyzing 
focus group data is to examine 
systematically the participants' 
comments in order to identify 
trends or patterns that recur 
among different individuals or 
groups. 



used. Thus, the analysis could focus primarily on addressing the 
information needs of pnogram decision-makers, yet be sufficiently 
flexible to identify issues that were not anticipated. 

The goal of analyzing focus group data is to examine systemati- 
cally the participants' comments in order to idv ' .tify trends or patterns 
that recur among different individuals or groups (Krueger, 1988). The 
outcome of the analysis may be a report that summarizes relevant 
trends, presents conclusions about the project's effectiveness, and 
offere suggestions for impn^vement. To accomplish these tasks, the 
analyst must sort through an array of statements and idendfy opinions, 
feelings, or perceptions that are common to several participants, even 
though they may be expressed in different words or styles. Some 
attention needs to be paid to determining the range and diversity of 
participants' perceptions. Less emphasis is given to singular opinions 
not supported by others i i the group. The use of numbers or percent- 
ages may help illustrate participants' responses, although they are not 
necessary for describing trends or supporting conclusions drawn from 
the analysis. 

A common misconception is that the analysis of focus group data 
is a simple process of finding a few good quotes that support the 
analyst's point of view. To produce valid conclusions, however, the 
analysis needs to be conducted systematically according to a pre- 
scribed sequence of procedures (see the three-stage process below). 
The process should allow staff to analyze a large amount of qualitative 
data and draw practical conclusions about the effectiveness of the 
project. 

Ideally, the person who conducts the focus group also should 
analyze the results. The results can be interpreted in part based on the 
first-hand experiences of the facilitator who actually witnessed the 
interpersonal dynamics, body language, and affective factors that 
influenced the activity. Sometimes, however, it may not be possible 
for individuals who collected the data to be involved in the analysis. 
For example, an instructor may conduct an interview with a small 
group of learners with the assistance of a bilingual staff member. 
Audio tapes of the interview, translated into English if necessary, 
might be reviewed and summarized by a professional evaluator for an 
evaluation report. Comments by those who conducted the focus group 
could be added to the transcript, providing the evaluator with valuable 
additional information. 

According to Krueger ^1988), the process of analyzing focus 
group data can be divided into three stages: (1) reviewing the raw 
data; (2) creating descriptive statements; and (3) interpreting the 
results. 

Reviewing the raw data. At this stage, staff members analyze the 
tapes or notes taken during the focus group. The raw data consist of 
the participants' actual statements recorded in writing or on audio or 
video tape. Also included in the raw data are the facilitator's descrip- 
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By probing specific top- 
ics, the facilitator can uncover 
causes of a problem and identify 
potential strategies for refining 
and improving the project. 



lions of the participants* such as their attitudes and body language and 
the general ambience of the session. 

During this stage, the analyst should focus on becoming as 
familiar as possible with the range of data collected and then begin 
organizing participants' responses by categories or topics. Data may 
be organised by (1) marking all related topics in the transcript with tlie 
same colored marking pen; (2) extracting comments that deal with the 
same issue and writing them on a single page; or (3) rearranging the 
text according to spec- nc categories. 

Creating descriptive statements. The next stage in the analysis 
involves reducing and organizing the raw data by creating descriptive 
statements that summarize the participants' responses according to 
topics. The summaiy also can include illustrative quotes that arc 
especially insightful or indicative of the participants' comments. 
Comments need not be transcribed verbatim; rather, staff may prefer 
to use alternative statements that capture tlie speakers' intended 
meaning. The analyst should begin, reducing the data and preparing the 
descriptive summary as soon as possible after each focus group 
session. 

Interpreting the results. At the final stage, the analyst needs to 
enhance the descriptive statements by drawing conclusions and 
interpretations from the data. The analyst should fonn conclusions 
about trends or patterns based on the purpose of the focus group 
session (for example, answers to staffs evaluation questions or 
evidence of progress toward meeting project objectives). Unexpected 
outcomes of the session also should be noted. To draw valid inferences 
from the data, the analyst should consider such factors as commonly 
used words or phrases, the frequency of certain responses, and the tone 
and intensity of the speaker's voice. It is also important to examine the 
context of participants* responses — that is, whether the comments 
occurred in response to a specific question or were spontaneously 
triggered by an open-ended question. 

The analyst may give more weight to comments based on some 
criteria; for example, statements based on personal experience may be 
accorded more importance than something the speaker heard someone 
else say. Other factors to consider are the consistency of particular 
statements and whether individuals change their minds based on 
others' comments. Analyzing such factors will enhance the validity 
and reliability of the findings. 

Using Focus Group Data 

Staff can use focus groups to gather information that is not easily 
obtained through written assessments. The interactive nature of the 
focus group process allows staff to shape the direction of the session 
and to delve into selected responses. By probing specific topics, the 
facilitator can uncover causes of a problem and identify potential 
strategies for refining and improving the project. Focus groups enable 
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staff to explore aspects of the project from the participants' perspec- 
tivcSt thereby enhancing tlie linkage between the learners' needs and 
the project's objectives. 

Focus group data can be used to identify new needs of partici- 
pants as well as clarify needs that were obtained earlier through 
surveys or individual interviews. By learning more about ilie partici- 
pants — their attitudes, the cultural barriers they face, their patterns of 
social interaction, and their levels of confidence — staff can make more 
infomicd choices about curriculum content, instructional approaches, 
and the overall design of the project. 

To use focus group data effectively, staff need to identify trends 
and patterns among the group's responses. In general, the significance 
of a given issue is determined by the frequency of comments about it. 
For example, a single comment such as "The book is too hard" would 
not carry as much weight in the data analysis as five or ten statements 
about the difficulty of the instructional materials. If several comments 
were made, \i\c staff might want to examine how appropriate the 
materials are for the learners' literacy levels. 

Staff also can use focus groups to obtain information about 
literacy-related events in the learners' families. For example, a prompt 
such as *Tell rne how you arc using the home learning kits with your 
children" may help staff detemine the effects of project activities on 
the learners' families. Comments may range from **Using the matcrijils 
weekly and enjoy them very much" (10 respondents) to "Not using the 
booklets because we don't have any crayons" (2 respondents). The 
first response helps confirm the success of tlie project; the second 
suggests that the staff need to help some parents obtain what is 
necessary to use the home leamii^g kits. 

Using focus groups enables staff to generate more detailed 
information than is possible through some other assessment proce- 
dures. For example, parenting skills, attitudes, confidence, and other 
hard-to-assess domains can be explored through focus groups. Be- 
cause feedback is gathered directly and spontaneously from the 
participants, results are often highly indicative of the participants' 
perspectives. 

Summarizing Focus Group Data In a Narrative Report 

Reports of focus groups should be designed according to the 
needs of the audience, the purpose of tlie report, and the goals and 
objectives of the project. The results of focus groups can be communi- 
cated orally or in writing. According to Krueger (1988), focus group 
reports can be written following three different models: 

1 . Raw data model which presents the question or topic that the 
focus group addressed and a list of the participants' responses; 

2. Summary description model which summarizes the results of the 
focus group and includes selected quotations from participants; 
or 
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3. Interpretive model which provides a descriptive summary, 
illusiraiivc quotes, and a narrative interpretation of the results. 

The raw data model is tl)c fastest and easiest to develop and is 
most appropriate for an audience that is interested in reviewing all of 
the participants' responses. This model, however, requires tJiat readers 
do their own analysis of the data. The descriptive and interpretive 
models are usually more effective because they provide examples of 
the data as well as detailed analyses of the findings. 

A focus group report should include a statement describing the 
purpose of the session, an outline of the key questions addressed by 
the group, and a summary of the way the session was set up (for ex- 
ample, number of people interviewed and method of selecting partici- 
pants). The results of the report should be organized and presented 
according to the key questions that the focus group addressed. It may 
be helpful to describe aspects of tJie session that may limit the useful- 
ness of the data or prevent conclusive statements about the project. A 
section for conclusions and recommendations is useful for integrating 
the key findings into a concise summary and providing suggestions for 
modifying the project based on the results. An appendix to the report 
may include the focus group protocol and additional quotes from Uic 
participants. 

Exam p!e 7 shows how to document the purpose, protocol, and 
results of a focus group session liiat addressed parenting workshops 
held during a FELP project. The sample narrative might appear in an 
annual evaluation report or final performance report. 



Exampli\7 ' / 



Sample Summary of Focus Group Results ; 

Tne purpose of the focus group was to determine the effec- ' 
tiveness of the five parenting wori<shops hekJ on Saturdays and 
the degree to which the workshops had met the needs of the 

adult learners and the objectives of the project. Thirty-five i 

learners partk^ipated in the focus group. j 

Protocol for the Focus Group I 

Staff asked the parents to describe the parenting wori<sf iops | 

they attended. Questtons included: Describe how you were j 

involved in selecting the toprcs. Were you unable to attend any | 

of the workshops? Why? Do you feel you are more confident as j 
a parent because of the wori^shops? Why? Do you help your 

children with their homework? What did you learn that you did ' 

not know before? What are you doing differently as a result of j 

the workshops? | 

At least half of the parents attended three or more of the five 
parenting workshops. They reported that ihey had an opportu- 
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Ilxample l (Continued) 



nity to work with staff to plan the workshops but were unable to 
recall any specific topics they had requested. All agreed that 
the workshops helped them as parents and enabled them to 
work more confidentiy with their children at home. A majority of 
parents indicated that they reviewed homework assignments, 
while only a small number actually helped their children with 
reading and writing assignments. 

The parents expressed general agreement about the ill 
effects of television viewing and indicated that they had been 
successful in reducing the amount of teieviston that their 
children watched at home. Most stated that they communicated 
with their chiWren in their native language, while onty two 
parents indicated that they interacted with their children in 
English on a rogular basis. The staff noted that, compared with 
the focus group heki earlier in the year, the parents in this 
sesston seemed more comnujnicative and confident when 
discussing parenting issues. 

In summary, the parents enjoyed the workshops and 
enthusiasttoally volunteered information during the sessions. 
Staff noted that parents who participated in both focus group 
sessions appeared more competent and confident when 
expressing their ideas in English than those wtio attended one 
session. Approximately half of the participants could respond in 
English while the other half required the assistance of an 
interpreter. In general, the workshops were well attended and 
seemed to provkJe parents with information that made them feel 
more confident alxut helping their children at home. Staff 
agreed that future workshops could be more effective by 
dealing with to^tCs that are (1) related to parent-child learning 
activities at home and (2) linked to the cunrculum used in 
weekly FELP classes. 



Observation Measures 

Observation measures can be used to collect and record various 
kinds of information. They may focus on such features as learners' 
characteristics, group interactions, or language and literacy perfor- 
mance. Observations may be used as closed-ended assessments, such 
as indicating tlie learner's level of listening comprehension. Or, they 
may be more open-ended as when staff members judge how confident 
a learner appears when participating in a cooperative learning group. 

This section deals with the collection, analysis, staff use, and 
reporting of data obtained with observation measures. References are 
made to the measures that appear in Chapter IV as Example 7 Gisten- 
ing and reading) and Example 8 (speaking and writing). A modified 
cohort evaluation design is presented to illustrate how projects can 
analyze assessment data in a manner consistent with ESEA, Title VII, 
requirements for comparison group data. 
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Colldcting ObMffvatio^^ '^^a wKh Rating Scales 

The observation meci^ures found in Chapter IV can be used Ik. 
collect data about the learners* progress in acquiring receptive Gistening 
and reading) and productive (speaking and writing) language skills. The 
instruments use rating scales for indicating the learner's degree of 
proficiency. 
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Observation Measure for 
Listening and Reading 



Typtof •mtK 


meat: Observaaon measure 






Purpoic: To assess listening and reading skills 






! 

j fl«c«pfr>« skm 


WW MM 


wm$otm 
support 


Withgrtt 


Notil 


Ustening 


Foltows fsmillMr instructions 












Follows n6w instructions 










FlMding 


Rttds /im/Z/j/- instructions 












Roads n«i¥ instructions i j 


1 



Example O 



Observation Measure for 
Speaking and Writing 



Type of iwiiiiiatr Observation measure 

t To assess speaking and writing skills 





Pfoductfve skjHM 


VV?fr>MM 


wm»ofm 


Whhgnat ! Nottt 
dnajity i St 


SpMtdng 


Produces fMm;!i^ri(x^v\ 
Instructions that are 
cl«ar and corppreh«n»ble 

, 1 


1 


1 






Produces ndtv spoken 
Instructions that are dear 
and comprehensible 






1 1 


1 


Wrtting 




Produces /am///ar written 
instructions ttiat are dear 
and comprehentiblt 


1 1 








Produces new written 
instructions that art dear 
and comprehenttMe 
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provide clear distinctions among 
developmental levels on the rat- 
ing scale and represent appropri- 
ate growth according to the 
project's literacy framework. 



The data collected with the observation measures can be used to 
make statements about the performance of an individual or group ac- 
cording to descriptors that are associated with each of the ratings, 
**With ease" "With some support " **With great difficulty" and **Not at 
all.'* Ttie instruments are depicted as examples that FELP staff can 
adapt to reflect the objectives of their project. Staff can modify the 
instruments by designing their own descriptors and establishing their 
own protocol for rating their learners in a consistent, reliable manner. 

The descriptors reflect staffs expectations about how learners 
will perform at various stages of development. The descriptors should 
provide clear distinctions among developmental levels on the rating 
scale and represent appropriate growth according to the project's lit- 
eracy framework (see chapters II and III for more on the literacy frame- 
woric). The lowest rating and its descriptor should reflect what the staff 
has determined to be the lowest level of language proficiency exhibited 
by new enroUees. The highest rating and its descriptor should indicate 
mastery or the highest level of proficiency that the staff consider attain- 
able for the leamers in the project. Each item on the scale should be 
mutually exclusive and part of an even number of selections (for ex- 
ample, four or six), thereby eliminating middle-ground options and 
requiring observers to make clear choices. Creating well-developed 
descriptors and training staff to use the observation measures will en- 
hance the reliability and validity of the instalments. 

Assessment through observation means watching leamers per- 
form various tasks related to the project. Such assessment can be done 
without having to set aside additional time in the project for testing. 
The observations should be based on the project's objectives and con- 
ducted systematically and consistently at predetermined time intervals. 

The following is a summary of strategies that FELP staff can use 
for obtaining accurate and meaningful data with observation measures. 

1. Establish descriptors that define different levels of performance 
that can be agreed upon by similariy qualified raters who are fa- 
miliar with the instructional design of the project. The descriptors 
should be (a) aligned with the curriculum and objectives of the 
project and (b) reflective of the lowest and highest perfonmance 
levels of the leamers. 

2. Create protocols and record observations at regular intervals (for 
example, monthly, quarteriy, or annually). The frequency is deter- 
mined by how the data will be used and the length of instructional 
units (lesson, course, or class). 

3. Develop ari instrument for collecting and managing the data such 
as the two observation measures shown earlier in this chapter. 

Methods for collecting observational data should ensure that the 
form for collecting the data is completed for each learner according to 
established intervals and that each learner's level of performance is 
recorded for each descriptor. The time intervals for assessment are 
determined by the techniques that will be used to analyze the results. 
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Data collected through 
observation measures may be 
analyzed according to a modified 
cohort design; that is, growth of 
project participants can be mea- 
sured against a comparison 
group (cohort), thereby satisfying 
one of the requirements of ESEA, 
Title VIL 



For example, a pre4)ost analysis requires that data be collected at the 
beginning and end of an instructional unit or specified period of time. 
To do a longitudinal analysis, staff need to collect data at more fre- 
quent intervals and over longer periods of time. Collecting data at fre- 
quent intervals for longitudinal analysis helps staff identify patterns 
and trends in the learners' performance. Such data are useful for com- 
paring the performance of the project participants with individuals who 
have similar characteristics or who have not received project services. 

Analyzing Observation Data with a Modified Cohort Design 

Staff can analyze data obtained with the observation measures by 
reviewing the results and determining the extent to which the learners 
are making progress in attaining project objectives. Data should be 
analyzed separately for each language group served by the project. 
Based on their analysis, staff can modify instructional activities to en- 
hance learner outcomes in the future. Additional analysis and interpre- 
tation can be conducted by assigning numerical values to the descrip- 
tors on the rating scale. Staff can report how many learners received 
what rating at each prescribed interval. If the number of learners dif- 
fers for each observation, the data can be presented as percentages of 
the total number of learners observed. Comparisons should be made 
through percentages because the number of learners usually varies for 
each year of the project and for each collection interval. 

Data collected through observation measures may be analyzed 
according to a modified cohort design; that is, growth of project partici- 
pants can be measured against a comparison group (cohort), thereby 
satisfying one of the requirements of ESEA, Title VII. This design is 
adapted from research by McConnell (1982). (See Chapter I for more 
on the comparison group requirement in ESEA, Title VII.) 

The modified cohort design provides an alternative to comparing 
the project participants' growth with adult learners who are not part of 
the project. If the cohort design is implemented at the beginning of the 
first year of a FELP project, staff would establish comparison group 
data by collecting data on new enrollees before instruction begins. As 
new learners enroll during the project, staff would add information 
about them to the comparison group data. At a predetermined interval, 
perhaps after completing an instructional unit, a semester, or the first 
year, staff would collect observational data and measure it against the 
comparison group data to assess the learners' progress toward meeting 
project objectives. For an extended longitudinal analysis, assessment 
data could be collected during the second or third year and measured 
against the comparison group data that had been compiled to that point. 

ESEA, Title VII, requirements specify that at least part of the 
evaluation be based on data collected at twelve-month intervals. Staff 
can meet this requirement by assessing project participants' perfor- 
mance after twelve months of instruction and comparing the results 
with the comparison group. Data on new enrollees who are added to 
the comparison group should be collected within two weeks of enroll- 
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ment, after the staff has become somewhat familiar with their back- 
ground. The data become comparison group data, collected whenever 
new learners enroll during the year, thereby accommodating open 
enrollment policies adopted by many FELP projects. 

The modified cohort design is based on the assumption that the 
comparison group data represent levels of performance that would 
have been demonstrated by individuals who had not benefitted from 
services of the FELP project — a population referred to in the ESEA, 
Title VII evaluation requirements as a **nonproject comparison group." 
(Additional assistance for implementing a modified cohort design is 
available from the Evaluation Assistance Centers funded by ESEA, 
Tide VII.) 

After the comparison group data have been established, staff can 
compare differences between the comparison group and the project 
participants in terms of the number and percentage of individuals who 
received each rating on the observation measures. Differences be- 
tween the two groups can be illustrated through bar or line graphs that 
identify each group according to time in the project (see Example 8, 
Table F, below). lUustrations should be accompanied by a narrative 
that describes the relationship between the learners' performance and 
the services provided by the project. 

The observation measures found in Chapter IV contain two 
perfomiance descriptors for each of the skills of listening, reading, 
speaking, and writing. By counting and reporting the number (or 
percentage) of learners observed performing at each level during each 
observation, staff can make global statements about the group's 
abilities in each skill. The observations may show that the learners 
were uniformly low in every skill at the beginning of instruction or 
that they exhibited great diversity in thieir abilities. Longitudinal or 
post-test data may show that most of the learners were performing at 
more proficient levels or that there was considerable variation in 
ability. 

The modified cohort design may show that learners who have 
participated in the project are performing at higher levels than mem- 
bers of the comparison group who have not yet benefited from pro- 
gram services, thereby validating the positive effects of the project. 
Longitudinal data collected over several years may reveal relation- 
ships between the learners' growth in performance and length of time 
they were enrolled in the project (see graphic illustration of such 
relationships in Example 8, Table F). 

Using the Results of Observation Measures 

To provide useful information, observation measures should be 
(1) tied direcUy to the objectives and instructional activities of the 
project and (2) conducted on a regular basis. By linking the descrip- 
tors and progression of ratings to instructional priorities, staff can 
obtain valuable data for assessing the learners* ongoing progress and 
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To make the best use of 
data collected with observation 
measureSr staff members need to 
examine individuaJ learners' per- 
formance and create aggregated 
profiles of the whole class. 



Like other alternative 
assessments, the purpose of 
observatk>n measures should be 
clearly established before an 
Instrument Is devek>ped or used. 



for improving the instructional program. ITie observation measures also 
can be used as initial assessment tools for determining learners* needs. 
For example, learners who are initially observed performing tasks 
"With ease** probably would need instruction that is different from 
those who are rated as **Not at all. * * 

Staff can use data obtained with observation measures for modify- 
ing curriculum content and instructional methodology. For instance, if 
the data indicate that many of the learners are performing listening 
tasks *'With ease** but writing assignments **With great difficulty,** staff 
may want to devote more time and effort to writing instruction. Simi- 
larly, if several lea.ners are observed performing a certain task **With 
great difficulty,** even after it has been presented and practiced many 
times, staff may want to consider how instructional activities could be 
modified to help learners gain mastery of the task. 

To make the best use of data collected with observation measures, 
staff members need to examine individual learners* performance and 
create aggregated profiles of the whole class. These findings can be 
discussed regularly at staff meetings to determine how the project can 
be more responsive to learners* needs. For example, learners whose 
individual data contain several ratings of "Not at all** or **With great 
difficulty** may have special needs that warrant more personal attention. 
If the aggregated data show ratings that are spread across many perfor- 
mance levels, staff members may want to discuss how they can address 
multi-level classes more effectively. 

Individual learners* performance records may be used to deter- 
mine whether current instructional groupings are appropriate and how 
new groups might be formed. Significant variation in ratings may be 
due to discrepancies in the staff members' knowledge of how to use the 
observation measures. Staff may need to clarify the protocols for using 
the measures and the definitions of tiie performance descriptors. 
Observation measures should be used systematically according to 
predetermined criteria that are related to the project*s objectives. 
During the observations, however, staff may identify other relevant 
information that should be documented. For example, staff may note 
that some learners are frustrated, socially isolated, or have auditory, 
speech, or visual impainnents. Staff also may document special talents 
and interests shown by learners. Such findings will help staff create 
activities that are more closely linked to learners* needs and make othrr 
improvements to tiie project. 

Like other alternative assessments, the purpose of observation 
measures should be clearly established before an instrument is devel- 
oped or used. Observation measures can be used for planning instruc- 
tion and placing learners in appropriate groups or activities. As dis- 
cussed in Chapter IV, observational data can provide staff with infor- 
mation about the learners' ongoing progress in attaining project objec- 
tives. Observation measures also can yield datp for the annual evalua- 
tion reports or the final performance report. 
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Open enrollment and un- 
even attendance in many FELP 
projects often prevent staff from 
assessing the same learners 
every time an observation is con- 
ducted. 



Reporting the Results of Observation IMeasures 

Reporting observational data involves summarizing the results of 
the observations and developing interpretations of the data. Data col- 
lected from observational measures can be used to make descriptive 
statements about what individual leamers or groups of learners can do 
relative to project goals, objectives, and instructional services. 

The observation measures found in Chapter IV are used to rate 
leamers according to their receptive (listening and reading) and produc- 
tive (speaking and writing) skills in handling new or familiar instruc- 
tions. What is new or familiar depends on the learners' prior experiences 
and the topics addressed in the FELP project Thus, the reporting of 
observation data should include descriptions or examples of the content 
or activities to which leamers were responding when the observation 
ratings were made. 

Bar graphs are especially useful for presenting summaries of 
observational data. For example, each bar could represent the number of 
leamers who performed a specific skill at a given level (that is, the 
number of leamers who read new instmctions "With great difficulty' '), 
The construction of the bar graph depends on what specific information 
the staff want to emphasize. Example 9 (Table G) shown below illus- 
trates growth in the learners' ability to produce written instmctions that 
are clear and comprehensible. 

Table G summarizes hypothetical data for a class of 25 participants 
who were rated at quarterly intervals. The table is followed by a narra- 
tive explanation. The numbers and narrative address only one of the 
eight areas assessed by the two observation measures. Staff could 
develop similar illustrations for data obtained for the other seven skill 
areas. When the numbers (n) vary greatly between groups of leamers or 
years of implementation, percentages can be used instead of numbers, 

Table G summarizes data collected on the same group of 25 
leamers. However, as was pointed out in Chapter I, open enrollment and 
uneven attendance in many FELP projects often prevent staff from 
assessing the same leamers every time an observation is conducted. 
Therefore, staff should explain this phenomenon in thie evaluation report 
and present assessment results only for the leamers who attended 
regularly. Staff also may want to include in the report some of the 
causes of attrition and the strategies that are being used to improve 
retention. 

If a project has open enrollment, a second quarterly observation 
conducted in one class may be the first observation for new enrollees 
and the second observation for experienced participants, In this case, 
data gathered on several leamers should be aggregated only if each set 
of data represents approximately equal amounts of instructional time. 
Data on leamers who started at different times may need to be reported 
separately, or if all learners' data are combined, the first observation 
should in fact be the first for each learner. 

Example 8 (Table F) illustrates the findings of a modified cohort 
design in which progress of leamers who have participated in the project 
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TABLE F I 

( 

Summary of Data Collected with Observation Measure (Modified Cohort Design) 

"Produces new written instructions that are clear and comprehensible" 




WHh E«M WW) Som« Support WHh Qrtct D»f5cu)ty Not At AH 



Sample Narrative: Evaluation results based on a modified cohort design indicate that the project had a 
positive impact on the development of literacy skills among adults. Furthermore, the data suggest that 
learners who have participated for two years show substantially greater gains than those who have been 
in the project for only one year. Staff suggested that differertces in performance between the two groups 
were attributed in part to the learners' needing a full year of participation in order to make more rapid 
progress in subsequent years. 



Example 9 



TABLE G 

Summary of Ratings of Four Observations (ns25) 

Quarterly Ratings on "Produces new written instructions that are dear and comprehensibl©" 



Rating 


S»ptfrH)*r 

First 
Observation 


Second 
Observation 


March 

Third 
Observation 


June 

Fourth 
Observation 


With Ease 


0 


0 


3 


10 


With Some Support 


0 


5 


12 


8 


With Great Difficulty 


5 


5 


10 


7 


Not at All 


20 


15 


0 


0 



Sample Narrative: Table G shows an increase In the percentage of higher raltnga from the first to the 
fourth observation and a decrease in the lesaixoficient ratings. For the first observation in Seplember, 
none of the learners was able to write new instructions that were clear and compr^ieneible. However, by 
the fourth observation in June, after nine months of instruction, 18 of the 25 (earners could write new 
instnjctions with ease or with some support. 



for one and two years is measured against the performance of a 
comparison group of learners who did not receive project services. 

Each of the eight receptive and productive skills identified on the 
two observation measures should be illustrated and reported sepa- 
rately. A table that illustrated all eight skills might confuse the reader 
and fail to delineate important details. Staff should include a narrative 
interpretation of the results with each table as shown in the examples 
in this section, or they could describe the results in narrative form and 
place it with the related goal or objective as shown in Example 10. 
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Report of Learners' Progress In Attaining Objective 
for Receptive and Productive Language Skills 

Objective - Literacy: By the end of each year of the project, 
partteipants will improve their receptive (listening and reading) 
and productive (speaking and writing) skills at least one level as 
measured by obsen/ation measures devetoped by the project. 

Sampis Narrative 

The objective was achieved during the second year of 
the project. Based on the observation measures, learners 
experienced growth in all eight literacy skill areas. The greatest 
growth occurred in listening skills with 72 percent of the learn - 
ers rated "With ease" for that category. Fifty-seven percent 
received the same rating for speaking, 29 percent for reading, 
and 16 percent for writing. The results of the observatkjn 
measures are consistent with the comments of staff who 
indk:atedthat listening and speaking skills were emphasized 
during the first two years of the project in response to the 
leamers' identified needs and requests. 

Based on the results of data analysis according to a 
modified cohort design, the project has had positive effects on 
the learners' literacy devek)pment. The language classes, 
which integrate literacy instruction with bask) life skills, repre- 
sent a potentially successful component of the FELP project. 



Summary 

Alternative approaches to assessment and evaluation are espe- 
cially useful for meeting the needs of participants in a FELP project. 
The results of alternative approaches can be used to (1) detemiine the 
extent to which a project is attaining its goals; (2) ascertain changes 
that will improve the overall effectiveness of a project; and (3) iden- 
tify successful components of a project. 



^ 

By considering authentic 
representations of what learners 
and staff are doing, alternative 
assessments can provide clear 
insights into the effects of the 
project on adult learners and 
their children. 



Staff can strengthen the use of alternative approaches by (1) 
collecting data systematically; (2) analyzing the data to provide 
meaningful interpretations of the results of assessments; (3) using the 
data to answer questions that staff, learners, and others have about the 
project; and (4) creating informative reports that summarize the results 
of the assessments. 

Surveys, performance samples, interviews, and observation 
measures are four kinds of alternative assessments which, together 
with standardized measures, can be used to create a comprehensive 
evaluation design for FELP projects. To maximize their effectiveness, 
alternative assessments need to be tailored to the unique goals, objec- 
tives, and program design of a FELP project. By considering authentic 
representations of what learners and staff are doing, alternative 
assessments can provide clear insights into the effects of the project on 
adult learners and their children. As an integral part of identifying 
needs, assessing ongoing progress, and determining project effective- 
ness, alternative approaches play an indispensable role in enabling 
staff and leamers to collaborate in designing a FELP project that is 
responsive to the needs of language-minority leamers, families, and 
communities. 



Selected References 

Krueger, Richard A. Focus Groups: A Practical Guide for Applied 
Research. Newbury Park, Calif.: Sage Publication!;, 1988. 

McConnell, Beverly B. "Evaluating Bilingual Education Using a Time 
Series Design," in New Directions for Program Evaluation: Applica- 
tions of Time Series Analysis to Evaluation, No. 16, Edited by Garlie 
A. Forehand. San Francisco, Calif.: Jossey-Bass, 1982, pp. 19-32. 

Patton, Michael Q. Qualitative Evaluation Methods. Beverly Hills, 
Calif.: Sage Publications, 1980. 



ERIC 



1 < ~ 

O ChapttrV ^ 



Assessing Success: 
FELP Resource Guide 



SURVEY OF USAGE 

Chapter V: Collecting, Analyzing and Reporting Alternative Assessment Results 

This resource guide must be helpful to you, the FELP director or evaluator, in order to be considered 
a success. In accepting the field test edition of the guide, we hope that you agree to participate in this 
usage survey. We plan to revise the guide based on your connnnents and criticisnns; it then will be 
published for a broader audience. Please share with us your critical connments. After reviewing the 
chapter, use the scale below to indicate whether the guide is written clearly; the scale indicates that 
it is 

1 . Clearly written and understandable, the format is clean; 

2. Generally clear, but some wordings or format could be improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at ail clear, not at all understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. For the information in this chapter, you probably will be working with your 
evaluator. In each of these areas, please use the scale to indicate whether you can now talk with your 
evaluator in an informed manner because the guide has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal; or 

4. Information that I don't need and won't use. 



Rate clarity Area to be rated Rate information 

1 2 3 4 Format of the Chapter Not applicable 

1 2 3 4 Wording of the chapter Notapplicable 

Not applicable The genera* ideas and information presented 12 3 4 
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Performance samples: 


Data collection 
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Data analysis 
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Staff utilization of results 
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Reporting the results 
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Interviews: 


Data collection 
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Data analysis 
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Staff utilization of results 
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Observation measures: 


Data collection 
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Data analysis 
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Reporting the results 
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Thank you for your assistance in this review of the guide. If there are other areas you would like to 
see added to this chapter, or other comments you would like to make, please feel free. 



^ Piease return your completed survey. You may fold, staple, and stamp this page - the address is on the back 
side. If you prefer, you may FAX it to Daniel D. Holt, Bilingual Education Office at (916} "^57-2928. We also 
would appreciate receiving a copy of any alternative assessments you create as a result of this guide. 




Fold here 




Daniel D. Holt 

Bilingual Education Office, California Department of Education 

PO Box 944272 
Sacranf)ento, CA 94244-2720 



Fold here 
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GLOSSARY 



Alternative assessment, A flexible assessment 
approach tliat is reprcsentaiivc of the curriculum, 
meaningful to learners, and indicative of tl>e 
learners* acquired ability or knowledge. 
Examples are surveys, interviews, observation 
measures, and perfonnance samples. 

Annual evaluation report. Submitted yearly to the 
Office of Bilingual Education and Minority 
Languages Affaire (OBEMLA), U.S. Department 
of Education, to summarize the accomplishments 
of a project. 

Assessment, The use of instruments and procedures 
to gather data on a regular basis. Assessment 
may focus on identifying learners' needs, 
documenting tlie learners* progress toward 
meeting iheir own goals, and ascertaining the 
extent to which the project objectives are being 
met. 

Baseline data. Data that are collected before project 
activities are implemented. For example, 
baseline data collected on new enroUces may be 
used for comparison with data obtained from 
learners at a later point in the project 

Descriptor. An explanation of an individual rating 
on a scale. For example, on a 0-5 scale for 
assessing reading, a descriptor for 0 may be 
*'Cannot read instructions" and the descriptor for 
5 might be **Reads instructions with ease.** 

English-as-a-second-language. English acquired by 
individuals whose native language is a language 
other than English. 

Evaluation. The process of integrating and 
analyzing assessment data at a given point in 
time for such purposes as developing and 
refining goals and objectives; documenting the 
learners* progress; and determining the overall 
success of the project. 



Family English Literacy Program (FEIJP), A 
program authorized in 1984 as part of Title VII 
of the Elementary and Secondary Education Act. 
FELP is defines as **a program of instruction 
designed to help lii"nited-English-proficient 
adults and out-of-school youth achieve 
competence in the English language. . . . Such 
programs may include instruction on how 
parents and family members can facilitate the 
educational achievement of limited-English- 
proficient children,** [20 USC 3283, 
Sec.7003(a)(7)] 

Final performance report. Submitted to the Office 
of Bilingual Education and Minority Languages 
Affairs (OBEMLA). U.S. Deparunent of 
Education, to summarize tlie accomplishments 
of a project during the entire grant period. 

Focus group. A group that is formed for the 
purpose of providing input. Group members are 
interviewed according to a protocol that has 
been previously established. 

Frequency count, A technique for tallying 
infonmaiion, usually from survey responses. A 
frequency count may provide information such 
as how many parents need child care or how 
often participants visit the library. 

Holistic scoring, A strategy for assessing 

performance by summarizing individual features 
to obtain an overall view of learners* progress or 
some other dimension of the project. 

Interview. An alternative assessment approach for 
obtaining detailed infonnation. An interview 
may be administered to individuals or to groups 
with respondents answering orally or in writing. 
Interviews may be used with focus groups. 

Literacy. The ability to use language and language- 
related strategies, especially those associated 
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with Reading and writing, in a variety of settings 
(family, community, school, woi-k) to achieve 
one*s own goals (personal, professional, 
academic). 

Literacy framework. Staff members* and learners' 
shared understanding of the leamers' literacy 
needs, program goals, curriculum and 
methodology, and approaches for assessing and 
evaluating literacy development. The elements 
of the framework should be coordinated so that 
they are interdependent and mutually- 
supportive. 

Literacy practices. The ways individuals use 
literacy in thieir daily lives. 

Modified cohort design. A method of analyzing 
data by comparing the perfomiance of project 
participants with comparable individuals who 
have not yet benefitted from project services. 

Norm-referenced assessment. See Standardized 
assessment. 

Objective. A dCvScription of an outcome or change 
that is expected as a result of implementing a 
project. The content of objectives is detennined 
by what the learners need, what the staff are 
prepared to teach, and what is possible to 
achieve within the time and budget constraints 
of the project. 

Observation measure. An altemative assessment 
instrument for obtaining various kinds of 
information. Observations may focus on such 
features as learners' characteristics, group 
interactions, or language and literacy 
performance. 

Portfolio. A collection of examples of a leamer's 
accomplishments such as reading logs, language 
experience stories, and writing samples. The 
contents of the portfolio can be used to show a 
learner's progress overtime. 

Performance sample. An example of a learner's 
work in selected tasks. Performance samples 
include writing samples, oral readings, role 
plays, and oral presentations. 



(Instructional) program. A coordinated set of 
activities that represent the curriculum and 
instructional components of a prcjeci. 

Project. Refers to all components to be 

implemented, including tiiose described in the 
instructional program. 

Protocol, A specified set of proced.ures and 

questions designed to achieve tJie purposes of an 
assessment activity. 

Rating scale. A technique for asse:5sing an area of 
interest (for example, reading ability or self- 
confidence) on a scale of intervals usually 
designated by numbers or descriptors. 

Reliable. Refers to the capacity of an assessment 
instrument or procedure to produce 
approximately the same results consistently on 
different occasions when the conditions of the 
assessment remain the same. For example, will 
the instrument provide the same results if 
learners are tested on Thursday instead of 
Wednesday, or if two or more competent judges 
scored the test independently? 

Standardized assessment. An instrument that is 
created to allow for comparisons between 
individuals' current achievement and the 
average perfonnance (norms) of selected 
participants (n^ -^^v ^ng group). An example is the 
Basic English H-h ills Test (BEST), available 
from the Center for Applied Linguistics, 
Washington, D.C. In this volume, the terms 
standardized assessment and norm-referenced 
assessment are used interchangeably. 

Survey. An altemative assessment approach for 
obtaining information. A survey may be 
administered to individuals or groups with 
respondents answering r.vdlly or in writing. 

Valid. Refers to the capacity of an assessment 
instrument or procedure to measure what it 
claims to measure. For example, does the 
reading test actually measure reading 
comprehension, or does success on the test also 
depend on the leamers* knowledge of U.S. 
culture? 
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